[PROPOSAL] NiFi for Incubation

[Joe Witt <jo...@gmail.com>]

Hello,


I would like to propose NiFi as an Apache Incubator Project.

In addition to the copy provided below the Wiki version of the
proposal can be found here:
http://wiki.apache.org/incubator/NiFiProposal

Thanks

Joe


= NiFi Proposal =

== Abstract ==
NiFi is a dataflow system based on the concepts of flow-based programming.

== Proposal ==
NiFi supports powerful and scalable directed graphs of data routing,
transformation, and system mediation logic.  Some of the high-level
capabilities and objectives of NiFi include:
  * Web-based user interface for seamless experience between design,
control, feedback, and monitoring of data flows
  * Highly configurable along several dimensions of quality of service
such as loss tolerant versus guaranteed delivery, low latency versus
high throughput, and priority based queuing
  * Fine-grained data provenance for all data received, forked,
joined, cloned, modified, sent, and ultimately dropped as data reaches
its configured end-state
  * Component-based extension model along well defined interfaces
enabling rapid development and effective testing

== Background ==
Reliable and effective dataflow between systems can be difficult
whether you're running scripts on a laptop or have a massive
distributed computing system operated by numerous teams and
organizations.  As the volume and rate of data grows and as the number
of systems, protocols, and formats increase and evolve so too does the
complexity and need for greater insight and agility.  These are the
dataflow challenges that NiFi was built to tackle.

NiFi is designed in a manner consistent with the core concepts
described in flow-based programming as originally documented by J.
Paul Morrison in the 1970s.  This model lends itself well to visual
diagramming, concurrency, componentization, testing, and reuse.  In
addition to staying close to the fundamentals of flow-based
programming, NiFi provides integration system specific features such
as: guaranteed delivery; back pressure; ability to gracefully handle
backlogs and data surges; and an operator interface that enables
on-the-fly data flow generation, modification, and observation.

== Rationale ==
NiFi provides a reliable, scalable, manageable and accountable
platform for developers and technical staff to create and evolve
powerful data flows.  Such a system is useful in many contexts
including large-scale enterprise integration, interaction with cloud
services and frameworks, business to business, intra-departmental, and
inter-departmental flows.  NiFi fits well within the Apache Software
Foundation (ASF) family as it depends on numerous ASF projects and
integrates with several others.  We also anticipate developing
extensions for several other ASF projects such as Cassandra, Kafka,
and Storm in the near future.

== Initial Goals ==
  * Ensure all dependencies are compliant with Apache License version
2.0 and all that all code and documentation artifacts have the correct
Apache licensing markings and notice.
  * Establish a formal release process and schedule, allowing for
dependable release cycles in a manner consistent with the Apache
development process.
  * Determine and establish a mechanism, possibly including a
sub-project construct, that allows for extensions to the core
application to occur at a pace that differs from the core application
itself.

== Current Status ==
=== Meritocracy ===
An integration platform is only as good as its ability to integrate
systems in a reliable, timely, and repeatable manner.  The same can be
said of its ability to attract talent and a variety of perspectives as
integration systems by their nature are always evolving.  We will
actively seek help and encourage promotion of influence in the project
through meritocracy.

=== Community ===
Over the past several years, NiFi has developed a strong community of
both developers and operators within the U.S. government.  We look
forward to helping grow this to a broader base of industries.

=== Core Developers ===
The initial core developers are employed by the National Security
Agency and defense contractors.  We will work to grow the community
among a more diverse set of developers and industries.

=== Alignment ===
>From its inception, NiFi was developed with an open source philosophy
in mind and with the hopes of eventually being truly open sourced.
The Apache way is consistent with the approach we have taken to date.
The ASF clearly provides a mature and effective environment for
successful development as is evident across the spectrum of well-known
projects.  Further, NiFi depends on numerous ASF libraries and
projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
HttpClient, Jakarta and Maven.  We also anticipate extensions and
dependencies with several more ASF projects, including Accumulo, Avro,
Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
This existing alignment with Apache and the desired community makes
the Apache Incubator a good fit for NiFi.

== Known Risks ==
=== Orphaned Products ===
Risk of orphaning is minimal.  The project user and developer base is
substantial, growing, and there is already extensive operational use
of NiFi.

=== Inexperience with Open Source ===
The initial committers to NiFi have limited experience with true open
source software development.  However, despite the project origins
being from closed source development we have modelled our behavior and
community development on The Apache Way to the greatest extent
possible.  This environment includes widely accessible source code
repositories, published artifacts, ticket tracking, and extensive
documentation. We also encourage contributions and frequent debate and
hold regular, collaborative discussions through e-mail, chat rooms,
and in-person meet-ups.  We are committed to the ideals of open source
software and will eagerly seek out mentors and sponsors who can help
us quickly come up to speed.

=== Homogenous Developers ===
The initial committers of NiFi come from a limited set of entities
though we are committed to recruiting and developing additional
committers from a broad spectrum of industries and backgrounds.

=== Reliance on Salaried Developers ===
We expect NiFi development to continue on salaried time and through
volunteer time.  The initial committers are paid by their employers to
contribute to this project.  We are committed to developing and
recruiting participation from developers both salaried and
non-salaried.

=== Relationship with other Apache Projects ===
As described in the alignment section, NiFi is already heavily
dependent on other ASF projects and we anticipate further dependence
and integration with new and emerging projects in the Apache family.

=== An Excessive Fascination with the Apache Brand ===
We respect the laudable Apache brand and that is certainly a factor in
the decision to propose NiFi for the Apache Incubator.  We intend to
provide a great deal of energy and capability to the ASF through this
project.  We will be sensitive to and respectful of any overuse of the
Apache brand and ensure our focus remains on how we benefit the Apache
community.

=== Documentation ===
At this time there is no NiFi documentation on the web.  However, we
have extensive documentation included within the application that
details usage of the many functions.  We will be rapidly expanding the
available documentation to cover things like installation, developer
guide, frequently asked questions, best practices, and more.  This
documentation will be posted to the NiFi wiki at apache.org.

== Initial Source ==
NiFi has been in active development since late 2006 with contributions
from dozens of developers and feedback from hundreds of users and
developers.  The core codebase is written in Java and includes
detailed Javadocs and feature documentation.

== Source and Intellectual Property Submission ==
Previously referred to as Niagarafiles, the NiFi code and
documentation materials will be submitted by the National Security
Agency.  NiFi has been developed by a mix of government employees and
private companies under government contract.  Material developed by
the government employees is in the public domain and no U.S. copyright
exists in works of the federal government.  For the contractor
developed material in the initial submission, the U.S. Government has
sufficient authority to open source per DFARS 252.227-7014.  NSA has
submitted the Software Grant Agreement and Corporate Contributor
License Agreement to the Apache Software Foundation.

== External Dependencies ==
We have at least one dependency on an LGPL library which we will
promptly address.  Otherwise, we believe all current dependencies are
compatible with the ASF guidelines.  Our dependency licenses come from
the following license styles:  Apache v 2.0, BSD, Public Domain,
Eclipse Public v1, MIT, CDDL v1.

== Cryptography ==
Consistent with http://www.apache.org/licenses/exports/ we believe
NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
cryptographic algorithms but is designed to use algorithms provided by
Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
These cryptographic algorithm providers are used to support SSL,
SSH/SFTP, and the encryption and decryption of sensitive properties.
In the event that it becomes necessary we will engage with appropriate
Apache members to ensure we file any necessary paperwork or clarified
any cryptographic export license concerns.

== Required Resources ==
=== Mailing Lists ===
  * user@nifi.incubator.apache.org
  * dev@nifi.incubator.apache.org
  * private@nifi.incubator.apache.org
  * commits@nifi.incubator.apache.org

=== Source Control ===
NiFi requests use of Git for source control
(git://git.apache.org/nifi.git).  We request a writeable Git repo for
NiFi with mirroring to be setup to Github through INFRA.  We request
sponsor Benson Margulies (bimargulies) to assist with creating the
INFRA ticket for this.

=== Issue Tracking ===
JIRA NiFi (NIFI)

=== Initial Committers ===
  * Brandon Devries <brandon.devries at gmail dot com>
  * Matt Gilman <matt.c.gilman at gmail dot com>
  * Tony Kurc <trkurc at gmail dot com>
  * Mark Payne <markap14 at hotmail dot com>
  * Adam Taft <adam at adamtaft dot com>
  * Joseph Witt <joewitt at gmail dot com>

=== Affiliations ===
  * Brandon Devries (Requitest, Inc.)
  * Matt Gilman (Raytheon)
  * Tony Kurc (National Security Agency)
  * Mark Payne (Sotera Defense Solutions, Inc.)
  * Adam Taft (Requitest, Inc.)
  * Joseph Witt (National Security Agency)

== Sponsors ==
=== Champion ===
  * Benson Margulies (Basis Technology) <bimargulies at apache dot org>

=== Nominated Mentors ===
  * Drew Farris (Booz Allen Hamilton) <drew at apache dot org>
  * Brock Noland (Cloudera) <brock at apache dot org>
  * Billie Rinaldi (Hortonworks) <billie at apache dot org>

=== Sponsoring Entity ===
We request the Apache Incubator to sponsor this project.

Re: [PROPOSAL] NiFi for Incubation

[Tim Williams <wi...@gmail.com>]

+1, good stuff...

--tim

On Wed, Nov 19, 2014 at 9:02 PM, Joe Witt <jo...@gmail.com> wrote:
> Hello,
>
>
> I would like to propose NiFi as an Apache Incubator Project.
>
> In addition to the copy provided below the Wiki version of the
> proposal can be found here:
> http://wiki.apache.org/incubator/NiFiProposal
>
> Thanks
>
> Joe
>
>
> = NiFi Proposal =
>
> == Abstract ==
> NiFi is a dataflow system based on the concepts of flow-based programming.
>
> == Proposal ==
> NiFi supports powerful and scalable directed graphs of data routing,
> transformation, and system mediation logic.  Some of the high-level
> capabilities and objectives of NiFi include:
>   * Web-based user interface for seamless experience between design,
> control, feedback, and monitoring of data flows
>   * Highly configurable along several dimensions of quality of service
> such as loss tolerant versus guaranteed delivery, low latency versus
> high throughput, and priority based queuing
>   * Fine-grained data provenance for all data received, forked,
> joined, cloned, modified, sent, and ultimately dropped as data reaches
> its configured end-state
>   * Component-based extension model along well defined interfaces
> enabling rapid development and effective testing
>
> == Background ==
> Reliable and effective dataflow between systems can be difficult
> whether you're running scripts on a laptop or have a massive
> distributed computing system operated by numerous teams and
> organizations.  As the volume and rate of data grows and as the number
> of systems, protocols, and formats increase and evolve so too does the
> complexity and need for greater insight and agility.  These are the
> dataflow challenges that NiFi was built to tackle.
>
> NiFi is designed in a manner consistent with the core concepts
> described in flow-based programming as originally documented by J.
> Paul Morrison in the 1970s.  This model lends itself well to visual
> diagramming, concurrency, componentization, testing, and reuse.  In
> addition to staying close to the fundamentals of flow-based
> programming, NiFi provides integration system specific features such
> as: guaranteed delivery; back pressure; ability to gracefully handle
> backlogs and data surges; and an operator interface that enables
> on-the-fly data flow generation, modification, and observation.
>
> == Rationale ==
> NiFi provides a reliable, scalable, manageable and accountable
> platform for developers and technical staff to create and evolve
> powerful data flows.  Such a system is useful in many contexts
> including large-scale enterprise integration, interaction with cloud
> services and frameworks, business to business, intra-departmental, and
> inter-departmental flows.  NiFi fits well within the Apache Software
> Foundation (ASF) family as it depends on numerous ASF projects and
> integrates with several others.  We also anticipate developing
> extensions for several other ASF projects such as Cassandra, Kafka,
> and Storm in the near future.
>
> == Initial Goals ==
>   * Ensure all dependencies are compliant with Apache License version
> 2.0 and all that all code and documentation artifacts have the correct
> Apache licensing markings and notice.
>   * Establish a formal release process and schedule, allowing for
> dependable release cycles in a manner consistent with the Apache
> development process.
>   * Determine and establish a mechanism, possibly including a
> sub-project construct, that allows for extensions to the core
> application to occur at a pace that differs from the core application
> itself.
>
> == Current Status ==
> === Meritocracy ===
> An integration platform is only as good as its ability to integrate
> systems in a reliable, timely, and repeatable manner.  The same can be
> said of its ability to attract talent and a variety of perspectives as
> integration systems by their nature are always evolving.  We will
> actively seek help and encourage promotion of influence in the project
> through meritocracy.
>
> === Community ===
> Over the past several years, NiFi has developed a strong community of
> both developers and operators within the U.S. government.  We look
> forward to helping grow this to a broader base of industries.
>
> === Core Developers ===
> The initial core developers are employed by the National Security
> Agency and defense contractors.  We will work to grow the community
> among a more diverse set of developers and industries.
>
> === Alignment ===
> From its inception, NiFi was developed with an open source philosophy
> in mind and with the hopes of eventually being truly open sourced.
> The Apache way is consistent with the approach we have taken to date.
> The ASF clearly provides a mature and effective environment for
> successful development as is evident across the spectrum of well-known
> projects.  Further, NiFi depends on numerous ASF libraries and
> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
> HttpClient, Jakarta and Maven.  We also anticipate extensions and
> dependencies with several more ASF projects, including Accumulo, Avro,
> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
> This existing alignment with Apache and the desired community makes
> the Apache Incubator a good fit for NiFi.
>
> == Known Risks ==
> === Orphaned Products ===
> Risk of orphaning is minimal.  The project user and developer base is
> substantial, growing, and there is already extensive operational use
> of NiFi.
>
> === Inexperience with Open Source ===
> The initial committers to NiFi have limited experience with true open
> source software development.  However, despite the project origins
> being from closed source development we have modelled our behavior and
> community development on The Apache Way to the greatest extent
> possible.  This environment includes widely accessible source code
> repositories, published artifacts, ticket tracking, and extensive
> documentation. We also encourage contributions and frequent debate and
> hold regular, collaborative discussions through e-mail, chat rooms,
> and in-person meet-ups.  We are committed to the ideals of open source
> software and will eagerly seek out mentors and sponsors who can help
> us quickly come up to speed.
>
> === Homogenous Developers ===
> The initial committers of NiFi come from a limited set of entities
> though we are committed to recruiting and developing additional
> committers from a broad spectrum of industries and backgrounds.
>
> === Reliance on Salaried Developers ===
> We expect NiFi development to continue on salaried time and through
> volunteer time.  The initial committers are paid by their employers to
> contribute to this project.  We are committed to developing and
> recruiting participation from developers both salaried and
> non-salaried.
>
> === Relationship with other Apache Projects ===
> As described in the alignment section, NiFi is already heavily
> dependent on other ASF projects and we anticipate further dependence
> and integration with new and emerging projects in the Apache family.
>
> === An Excessive Fascination with the Apache Brand ===
> We respect the laudable Apache brand and that is certainly a factor in
> the decision to propose NiFi for the Apache Incubator.  We intend to
> provide a great deal of energy and capability to the ASF through this
> project.  We will be sensitive to and respectful of any overuse of the
> Apache brand and ensure our focus remains on how we benefit the Apache
> community.
>
> === Documentation ===
> At this time there is no NiFi documentation on the web.  However, we
> have extensive documentation included within the application that
> details usage of the many functions.  We will be rapidly expanding the
> available documentation to cover things like installation, developer
> guide, frequently asked questions, best practices, and more.  This
> documentation will be posted to the NiFi wiki at apache.org.
>
> == Initial Source ==
> NiFi has been in active development since late 2006 with contributions
> from dozens of developers and feedback from hundreds of users and
> developers.  The core codebase is written in Java and includes
> detailed Javadocs and feature documentation.
>
> == Source and Intellectual Property Submission ==
> Previously referred to as Niagarafiles, the NiFi code and
> documentation materials will be submitted by the National Security
> Agency.  NiFi has been developed by a mix of government employees and
> private companies under government contract.  Material developed by
> the government employees is in the public domain and no U.S. copyright
> exists in works of the federal government.  For the contractor
> developed material in the initial submission, the U.S. Government has
> sufficient authority to open source per DFARS 252.227-7014.  NSA has
> submitted the Software Grant Agreement and Corporate Contributor
> License Agreement to the Apache Software Foundation.
>
> == External Dependencies ==
> We have at least one dependency on an LGPL library which we will
> promptly address.  Otherwise, we believe all current dependencies are
> compatible with the ASF guidelines.  Our dependency licenses come from
> the following license styles:  Apache v 2.0, BSD, Public Domain,
> Eclipse Public v1, MIT, CDDL v1.
>
> == Cryptography ==
> Consistent with http://www.apache.org/licenses/exports/ we believe
> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
> cryptographic algorithms but is designed to use algorithms provided by
> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
> These cryptographic algorithm providers are used to support SSL,
> SSH/SFTP, and the encryption and decryption of sensitive properties.
> In the event that it becomes necessary we will engage with appropriate
> Apache members to ensure we file any necessary paperwork or clarified
> any cryptographic export license concerns.
>
> == Required Resources ==
> === Mailing Lists ===
>   * user@nifi.incubator.apache.org
>   * dev@nifi.incubator.apache.org
>   * private@nifi.incubator.apache.org
>   * commits@nifi.incubator.apache.org
>
> === Source Control ===
> NiFi requests use of Git for source control
> (git://git.apache.org/nifi.git).  We request a writeable Git repo for
> NiFi with mirroring to be setup to Github through INFRA.  We request
> sponsor Benson Margulies (bimargulies) to assist with creating the
> INFRA ticket for this.
>
> === Issue Tracking ===
> JIRA NiFi (NIFI)
>
> === Initial Committers ===
>   * Brandon Devries <brandon.devries at gmail dot com>
>   * Matt Gilman <matt.c.gilman at gmail dot com>
>   * Tony Kurc <trkurc at gmail dot com>
>   * Mark Payne <markap14 at hotmail dot com>
>   * Adam Taft <adam at adamtaft dot com>
>   * Joseph Witt <joewitt at gmail dot com>
>
> === Affiliations ===
>   * Brandon Devries (Requitest, Inc.)
>   * Matt Gilman (Raytheon)
>   * Tony Kurc (National Security Agency)
>   * Mark Payne (Sotera Defense Solutions, Inc.)
>   * Adam Taft (Requitest, Inc.)
>   * Joseph Witt (National Security Agency)
>
> == Sponsors ==
> === Champion ===
>   * Benson Margulies (Basis Technology) <bimargulies at apache dot org>
>
> === Nominated Mentors ===
>   * Drew Farris (Booz Allen Hamilton) <drew at apache dot org>
>   * Brock Noland (Cloudera) <brock at apache dot org>
>   * Billie Rinaldi (Hortonworks) <billie at apache dot org>
>
> === Sponsoring Entity ===
> We request the Apache Incubator to sponsor this project.

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [PROPOSAL] NiFi for Incubation

[jan i <ja...@apache.org>]

Hi Joe.

I would really like to welcome this project at ASF.

I have one slight concern, could you be a bit more specific about the
diversity of the initial committers ?

I could read the proposal as if they come from one or more very related
entities.

I would also stresss in "initial goals", that the project should get
committers from outside of the circles you mention, to ensure diversity in
the project.

rgds
jan i


On 20 November 2014 06:27, Brock Noland <br...@cloudera.com> wrote:

> Hi Joe,
>
> I know you've done a tremendous amount of work to make this happen so I am
> extremely happy this is *finally* making it's way to the incubator!
>
> I look forward to helping in anyway I can.
>
> Cheers!
> Brock
>
> On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > This is *fan freakin¹ tastic* Sounds like an awesome  project and
> > glad to hear a relationship to Tika! Awesome to see more government
> > projects coming into the ASF!
> >
> > you already have a great set of mentors and I don¹t really have more
> > time on my plate, but really happy and will try and monitor and help
> > on the lists.
> >
> > Cheers!
> >
> > Chris
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Chief Architect
> > Instrument Software and Science Data Systems Section (398)
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 168-519, Mailstop: 168-527
> > Email: chris.a.mattmann@nasa.gov
> > WWW:  http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Associate Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Joe Witt <jo...@gmail.com>
> > Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> > Date: Thursday, November 20, 2014 at 3:02 AM
> > To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> > Subject: [PROPOSAL] NiFi for Incubation
> >
> > >Hello,
> > >
> > >
> > >I would like to propose NiFi as an Apache Incubator Project.
> > >
> > >In addition to the copy provided below the Wiki version of the
> > >proposal can be found here:
> > >http://wiki.apache.org/incubator/NiFiProposal
> > >
> > >Thanks
> > >
> > >Joe
> > >
> > >
> > >= NiFi Proposal =
> > >
> > >== Abstract ==
> > >NiFi is a dataflow system based on the concepts of flow-based
> programming.
> > >
> > >== Proposal ==
> > >NiFi supports powerful and scalable directed graphs of data routing,
> > >transformation, and system mediation logic.  Some of the high-level
> > >capabilities and objectives of NiFi include:
> > >  * Web-based user interface for seamless experience between design,
> > >control, feedback, and monitoring of data flows
> > >  * Highly configurable along several dimensions of quality of service
> > >such as loss tolerant versus guaranteed delivery, low latency versus
> > >high throughput, and priority based queuing
> > >  * Fine-grained data provenance for all data received, forked,
> > >joined, cloned, modified, sent, and ultimately dropped as data reaches
> > >its configured end-state
> > >  * Component-based extension model along well defined interfaces
> > >enabling rapid development and effective testing
> > >
> > >== Background ==
> > >Reliable and effective dataflow between systems can be difficult
> > >whether you're running scripts on a laptop or have a massive
> > >distributed computing system operated by numerous teams and
> > >organizations.  As the volume and rate of data grows and as the number
> > >of systems, protocols, and formats increase and evolve so too does the
> > >complexity and need for greater insight and agility.  These are the
> > >dataflow challenges that NiFi was built to tackle.
> > >
> > >NiFi is designed in a manner consistent with the core concepts
> > >described in flow-based programming as originally documented by J.
> > >Paul Morrison in the 1970s.  This model lends itself well to visual
> > >diagramming, concurrency, componentization, testing, and reuse.  In
> > >addition to staying close to the fundamentals of flow-based
> > >programming, NiFi provides integration system specific features such
> > >as: guaranteed delivery; back pressure; ability to gracefully handle
> > >backlogs and data surges; and an operator interface that enables
> > >on-the-fly data flow generation, modification, and observation.
> > >
> > >== Rationale ==
> > >NiFi provides a reliable, scalable, manageable and accountable
> > >platform for developers and technical staff to create and evolve
> > >powerful data flows.  Such a system is useful in many contexts
> > >including large-scale enterprise integration, interaction with cloud
> > >services and frameworks, business to business, intra-departmental, and
> > >inter-departmental flows.  NiFi fits well within the Apache Software
> > >Foundation (ASF) family as it depends on numerous ASF projects and
> > >integrates with several others.  We also anticipate developing
> > >extensions for several other ASF projects such as Cassandra, Kafka,
> > >and Storm in the near future.
> > >
> > >== Initial Goals ==
> > >  * Ensure all dependencies are compliant with Apache License version
> > >2.0 and all that all code and documentation artifacts have the correct
> > >Apache licensing markings and notice.
> > >  * Establish a formal release process and schedule, allowing for
> > >dependable release cycles in a manner consistent with the Apache
> > >development process.
> > >  * Determine and establish a mechanism, possibly including a
> > >sub-project construct, that allows for extensions to the core
> > >application to occur at a pace that differs from the core application
> > >itself.
> > >
> > >== Current Status ==
> > >=== Meritocracy ===
> > >An integration platform is only as good as its ability to integrate
> > >systems in a reliable, timely, and repeatable manner.  The same can be
> > >said of its ability to attract talent and a variety of perspectives as
> > >integration systems by their nature are always evolving.  We will
> > >actively seek help and encourage promotion of influence in the project
> > >through meritocracy.
> > >
> > >=== Community ===
> > >Over the past several years, NiFi has developed a strong community of
> > >both developers and operators within the U.S. government.  We look
> > >forward to helping grow this to a broader base of industries.
> > >
> > >=== Core Developers ===
> > >The initial core developers are employed by the National Security
> > >Agency and defense contractors.  We will work to grow the community
> > >among a more diverse set of developers and industries.
> > >
> > >=== Alignment ===
> > >From its inception, NiFi was developed with an open source philosophy
> > >in mind and with the hopes of eventually being truly open sourced.
> > >The Apache way is consistent with the approach we have taken to date.
> > >The ASF clearly provides a mature and effective environment for
> > >successful development as is evident across the spectrum of well-known
> > >projects.  Further, NiFi depends on numerous ASF libraries and
> > >projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
> > >HttpClient, Jakarta and Maven.  We also anticipate extensions and
> > >dependencies with several more ASF projects, including Accumulo, Avro,
> > >Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
> > >This existing alignment with Apache and the desired community makes
> > >the Apache Incubator a good fit for NiFi.
> > >
> > >== Known Risks ==
> > >=== Orphaned Products ===
> > >Risk of orphaning is minimal.  The project user and developer base is
> > >substantial, growing, and there is already extensive operational use
> > >of NiFi.
> > >
> > >=== Inexperience with Open Source ===
> > >The initial committers to NiFi have limited experience with true open
> > >source software development.  However, despite the project origins
> > >being from closed source development we have modelled our behavior and
> > >community development on The Apache Way to the greatest extent
> > >possible.  This environment includes widely accessible source code
> > >repositories, published artifacts, ticket tracking, and extensive
> > >documentation. We also encourage contributions and frequent debate and
> > >hold regular, collaborative discussions through e-mail, chat rooms,
> > >and in-person meet-ups.  We are committed to the ideals of open source
> > >software and will eagerly seek out mentors and sponsors who can help
> > >us quickly come up to speed.
> > >
> > >=== Homogenous Developers ===
> > >The initial committers of NiFi come from a limited set of entities
> > >though we are committed to recruiting and developing additional
> > >committers from a broad spectrum of industries and backgrounds.
> > >
> > >=== Reliance on Salaried Developers ===
> > >We expect NiFi development to continue on salaried time and through
> > >volunteer time.  The initial committers are paid by their employers to
> > >contribute to this project.  We are committed to developing and
> > >recruiting participation from developers both salaried and
> > >non-salaried.
> > >
> > >=== Relationship with other Apache Projects ===
> > >As described in the alignment section, NiFi is already heavily
> > >dependent on other ASF projects and we anticipate further dependence
> > >and integration with new and emerging projects in the Apache family.
> > >
> > >=== An Excessive Fascination with the Apache Brand ===
> > >We respect the laudable Apache brand and that is certainly a factor in
> > >the decision to propose NiFi for the Apache Incubator.  We intend to
> > >provide a great deal of energy and capability to the ASF through this
> > >project.  We will be sensitive to and respectful of any overuse of the
> > >Apache brand and ensure our focus remains on how we benefit the Apache
> > >community.
> > >
> > >=== Documentation ===
> > >At this time there is no NiFi documentation on the web.  However, we
> > >have extensive documentation included within the application that
> > >details usage of the many functions.  We will be rapidly expanding the
> > >available documentation to cover things like installation, developer
> > >guide, frequently asked questions, best practices, and more.  This
> > >documentation will be posted to the NiFi wiki at apache.org.
> > >
> > >== Initial Source ==
> > >NiFi has been in active development since late 2006 with contributions
> > >from dozens of developers and feedback from hundreds of users and
> > >developers.  The core codebase is written in Java and includes
> > >detailed Javadocs and feature documentation.
> > >
> > >== Source and Intellectual Property Submission ==
> > >Previously referred to as Niagarafiles, the NiFi code and
> > >documentation materials will be submitted by the National Security
> > >Agency.  NiFi has been developed by a mix of government employees and
> > >private companies under government contract.  Material developed by
> > >the government employees is in the public domain and no U.S. copyright
> > >exists in works of the federal government.  For the contractor
> > >developed material in the initial submission, the U.S. Government has
> > >sufficient authority to open source per DFARS 252.227-7014.  NSA has
> > >submitted the Software Grant Agreement and Corporate Contributor
> > >License Agreement to the Apache Software Foundation.
> > >
> > >== External Dependencies ==
> > >We have at least one dependency on an LGPL library which we will
> > >promptly address.  Otherwise, we believe all current dependencies are
> > >compatible with the ASF guidelines.  Our dependency licenses come from
> > >the following license styles:  Apache v 2.0, BSD, Public Domain,
> > >Eclipse Public v1, MIT, CDDL v1.
> > >
> > >== Cryptography ==
> > >Consistent with http://www.apache.org/licenses/exports/ we believe
> > >NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
> > >cryptographic algorithms but is designed to use algorithms provided by
> > >Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
> > >These cryptographic algorithm providers are used to support SSL,
> > >SSH/SFTP, and the encryption and decryption of sensitive properties.
> > >In the event that it becomes necessary we will engage with appropriate
> > >Apache members to ensure we file any necessary paperwork or clarified
> > >any cryptographic export license concerns.
> > >
> > >== Required Resources ==
> > >=== Mailing Lists ===
> > >  * user@nifi.incubator.apache.org
> > >  * dev@nifi.incubator.apache.org
> > >  * private@nifi.incubator.apache.org
> > >  * commits@nifi.incubator.apache.org
> > >
> > >=== Source Control ===
> > >NiFi requests use of Git for source control
> > >(git://git.apache.org/nifi.git).  We request a writeable Git repo for
> > >NiFi with mirroring to be setup to Github through INFRA.  We request
> > >sponsor Benson Margulies (bimargulies) to assist with creating the
> > >INFRA ticket for this.
> > >
> > >=== Issue Tracking ===
> > >JIRA NiFi (NIFI)
> > >
> > >=== Initial Committers ===
> > >  * Brandon Devries <brandon.devries at gmail dot com>
> > >  * Matt Gilman <matt.c.gilman at gmail dot com>
> > >  * Tony Kurc <trkurc at gmail dot com>
> > >  * Mark Payne <markap14 at hotmail dot com>
> > >  * Adam Taft <adam at adamtaft dot com>
> > >  * Joseph Witt <joewitt at gmail dot com>
> > >
> > >=== Affiliations ===
> > >  * Brandon Devries (Requitest, Inc.)
> > >  * Matt Gilman (Raytheon)
> > >  * Tony Kurc (National Security Agency)
> > >  * Mark Payne (Sotera Defense Solutions, Inc.)
> > >  * Adam Taft (Requitest, Inc.)
> > >  * Joseph Witt (National Security Agency)
> > >
> > >== Sponsors ==
> > >=== Champion ===
> > >  * Benson Margulies (Basis Technology) <bimargulies at apache dot org>
> > >
> > >=== Nominated Mentors ===
> > >  * Drew Farris (Booz Allen Hamilton) <drew at apache dot org>
> > >  * Brock Noland (Cloudera) <brock at apache dot org>
> > >  * Billie Rinaldi (Hortonworks) <billie at apache dot org>
> > >
> > >=== Sponsoring Entity ===
> > >We request the Apache Incubator to sponsor this project.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>

Re: [PROPOSAL] NiFi for Incubation

[Joe Witt <jo...@gmail.com>]

Andrew - I've updated the proposal to reflect you as a mentor.

  * Andrew Purtell (Salesforce) <apurtell at apache dot org>, IPMC Member

Thanks again

Joe

On Fri, Nov 21, 2014 at 5:29 PM, Joe Witt <jo...@gmail.com> wrote:

> Andrew - appreciate your comments and offer to assist as a mentor.  Now
> that the vote for incubation has kicked off i'm not sure what i'm allowed
> to do or not.  Will check will Benson.
>
> For what it is worth I agree Sean raised a great point.  I think it will
> be an important part of our community building effort.  And to echo Josh's
> point this proposal represents not just our intent to grow the community
> outside our current home but rather move homes - to the ASF.  We're very
> excited to have this opportunity.
>
> Thanks
> Joe
>
> On Fri, Nov 21, 2014 at 5:19 PM, Josh Elser <el...@apache.org> wrote:
>
>> Thanks for the clarification -- that wasn't how it came across to me.
>>
>>
>> Andrew Purtell wrote:
>>
>>> I didn't make an assertion *about* Accumulo, what I said exactly was:
>>>
>>>          "that Accumulo may or may not suffer from "bifurcation" is
>>> debatable"
>>>
>>> I assert there are two sides to that argument (I've heard them both)​​,
>>> merely to suggest that Sean's concerns should not be dismissed as they
>>> were.
>>>
>>>
>>> On Fri, Nov 21, 2014 at 2:11 PM, Josh Elser<el...@apache.org>  wrote:
>>>
>>>  I would tend to disagree with your assertion about Accumulo, but this is
>>>> neither the time nor the place to discuss Accumulo's strengths and
>>>> weaknesses.
>>>>
>>>> I'll make the assumption that request to enter incubation is a strong
>>>> intent to grow outside of the project's current home.
>>>>
>>>> +1 (non-binding) for entry
>>>>
>>>>
>>>> Andrew Purtell wrote:
>>>>
>>>>     From what I understand, that Accumulo may or may not suffer from
>>>>> "bifurcation" is debatable despite this blanket claim otherwise.
>>>>>
>>>>> Regardless, Sean's point is valid, even if only a theoretical concern
>>>>> for
>>>>> NiFi at this point. If the podling is interested in growing outside
>>>>> community I'd be happy to sign on as a mentor to help that happen.
>>>>>
>>>>> On Thu, Nov 20, 2014 at 3:32 AM, Benson Margulies<bimargulies@gmail.
>>>>> com>
>>>>> wrote:
>>>>>
>>>>>   Sean,
>>>>>
>>>>>> The precedent of Accumulo is that the govt people and agencies
>>>>>> involved
>>>>>> are
>>>>>> ready and able to have their staff collaborate openly in an Apache
>>>>>> community. There's no need to contemplate bifurcation; we have this
>>>>>> proposal because the management recognizes that this collaboration
>>>>>> produces
>>>>>> better stuff that solves more problems than the 'inside the tent'
>>>>>> alternative.
>>>>>>
>>>>>> --benson
>>>>>>
>>>>>>
>>>>>> On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey<bu...@cloudera.com>
>>>>>> wrote:
>>>>>>
>>>>>>   I'm really excited to see NiFi come to the incubator; it'd be a
>>>>>> great
>>>>>>
>>>>>>> addition to the ASF.
>>>>>>>
>>>>>>> A few points in the proposal:
>>>>>>>
>>>>>>>   == Initial Goals ==
>>>>>>>
>>>>>>>> One of these should be to grow the community outside of the current
>>>>>>>>
>>>>>>> niche,
>>>>>>
>>>>>>  IMHO.
>>>>>>>
>>>>>>> More on this below under "orphaned projects"
>>>>>>>
>>>>>>>     * Determine and establish a mechanism, possibly including a
>>>>>>>
>>>>>>>> sub-project construct, that allows for extensions to the core
>>>>>>>>> application to occur at a pace that differs from the core
>>>>>>>>> application
>>>>>>>>> itself.
>>>>>>>>>
>>>>>>>>>  I don't think the proposal needs to include the "e.g. with
>>>>>>>>
>>>>>>> sub-projects"
>>>>>>> part. Just noting
>>>>>>> that your goals in the incubator are to address the need to have
>>>>>>>
>>>>>>>  different
>>>>>>
>>>>>>  release cycles
>>>>>>> for core and extensions is sufficient.
>>>>>>>
>>>>>>>
>>>>>>>   === Community ===
>>>>>>>
>>>>>>>> Over the past several years, NiFi has developed a strong community
>>>>>>>>> of
>>>>>>>>> both developers and operators within the U.S. government.  We look
>>>>>>>>> forward to helping grow this to a broader base of industries.
>>>>>>>>>
>>>>>>>>>   How much, if any, of this community do you expect to engage via
>>>>>>>>> the
>>>>>>>>>
>>>>>>>> customary project
>>>>>>> lists once NiFi is established within the ASF? Will the project be
>>>>>>> able
>>>>>>>
>>>>>>>  to
>>>>>>
>>>>>>  leverage this
>>>>>>> established group?
>>>>>>>
>>>>>>>
>>>>>>>   === Orphaned Products ===
>>>>>>>
>>>>>>>> Risk of orphaning is minimal.  The project user and developer base
>>>>>>>>> is
>>>>>>>>> substantial, growing, and there is already extensive operational
>>>>>>>>> use
>>>>>>>>> of NiFi.
>>>>>>>>>
>>>>>>>>>  Given that the established base is internal to the U.S.
>>>>>>>> government, I'd
>>>>>>>>
>>>>>>> encourage the
>>>>>>> podling to consider the risk of a bifurcated project should a
>>>>>>> substantial
>>>>>>> outside
>>>>>>> community fail to emerge or if those internal users should fail to
>>>>>>> engage
>>>>>>> with the
>>>>>>> outside community.
>>>>>>>
>>>>>>> You cover a related issue in your "Homogenous Developers" section.
>>>>>>> But I
>>>>>>> think
>>>>>>> building on the "Community" section of the current state to call this
>>>>>>> out
>>>>>>> as an
>>>>>>> independent issue is worthwhile.
>>>>>>>
>>>>>>>
>>>>>>>   possible.  This environment includes widely accessible source code
>>>>>>>
>>>>>>>> repositories, published artifacts, ticket tracking, and extensive
>>>>>>>>> documentation. We also encourage contributions and frequent debate
>>>>>>>>> and
>>>>>>>>> hold regular, collaborative discussions through e-mail, chat rooms,
>>>>>>>>> and in-person meet-ups.
>>>>>>>>>
>>>>>>>>>  Do you anticipate any difficulties moving these established
>>>>>>>>
>>>>>>> communication
>>>>>>> mechanisms to ASF public lists?
>>>>>>>
>>>>>>>   === Documentation ===
>>>>>>>
>>>>>>>> At this time there is no NiFi documentation on the web.  However, we
>>>>>>>>> have extensive documentation included within the application that
>>>>>>>>> details usage of the many functions.  We will be rapidly expanding
>>>>>>>>> the
>>>>>>>>> available documentation to cover things like installation,
>>>>>>>>> developer
>>>>>>>>> guide, frequently asked questions, best practices, and more.  This
>>>>>>>>> documentation will be posted to the NiFi wiki at apache.org.
>>>>>>>>>
>>>>>>>>>  I love projects that start with documentation. :)
>>>>>>>>
>>>>>>> I don't think the proposal needs to include that the documentation
>>>>>>> will
>>>>>>>
>>>>>>>  be
>>>>>>
>>>>>>  posted
>>>>>>> to the NiFi wiki, since that's an implementation detail. Just say
>>>>>>> "this
>>>>>>> documentation
>>>>>>> will be made available via the NiFi project's use of incubator
>>>>>>> infra."
>>>>>>>
>>>>>>> (I'll save detail for the eventual dev@ list, but you should
>>>>>>> strongly
>>>>>>> consider not
>>>>>>> using the wiki to host this documentation.)
>>>>>>>
>>>>>>> -Sean
>>>>>>>
>>>>>>> On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland<br...@cloudera.com>
>>>>>>>
>>>>>>>  wrote:
>>>>>>
>>>>>>  Hi Joe,
>>>>>>>
>>>>>>>> I know you've done a tremendous amount of work to make this happen
>>>>>>>> so I
>>>>>>>>
>>>>>>>>  am
>>>>>>>
>>>>>>>  extremely happy this is *finally* making it's way to the incubator!
>>>>>>>>
>>>>>>>> I look forward to helping in anyway I can.
>>>>>>>>
>>>>>>>> Cheers!
>>>>>>>> Brock
>>>>>>>>
>>>>>>>> On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980)<
>>>>>>>> chris.a.mattmann@jpl.nasa.gov>   wrote:
>>>>>>>>
>>>>>>>>   This is *fan freakin¹ tastic* Sounds like an awesome  project and
>>>>>>>>
>>>>>>>>> glad to hear a relationship to Tika! Awesome to see more government
>>>>>>>>> projects coming into the ASF!
>>>>>>>>>
>>>>>>>>> you already have a great set of mentors and I don¹t really have
>>>>>>>>> more
>>>>>>>>> time on my plate, but really happy and will try and monitor and
>>>>>>>>> help
>>>>>>>>> on the lists.
>>>>>>>>>
>>>>>>>>> Cheers!
>>>>>>>>>
>>>>>>>>> Chris
>>>>>>>>>
>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>>> Chief Architect
>>>>>>>>> Instrument Software and Science Data Systems Section (398)
>>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>>> Office: 168-519, Mailstop: 168-527
>>>>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>>>>> WWW:  http://sunset.usc.edu/~mattmann/
>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>> Adjunct Associate Professor, Computer Science Department
>>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Joe Witt<jo...@gmail.com>
>>>>>>>>> Reply-To: "general@incubator.apache.org"<
>>>>>>>>>
>>>>>>>>>  general@incubator.apache.org
>>>>>>>>
>>>>>>> Date: Thursday, November 20, 2014 at 3:02 AM
>>>>>>>
>>>>>>>> To: "general@incubator.apache.org"<ge...@incubator.apache.org>
>>>>>>>>> Subject: [PROPOSAL] NiFi for Incubation
>>>>>>>>>
>>>>>>>>>   Hello,
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I would like to propose NiFi as an Apache Incubator Project.
>>>>>>>>>>
>>>>>>>>>> In addition to the copy provided below the Wiki version of the
>>>>>>>>>> proposal can be found here:
>>>>>>>>>> http://wiki.apache.org/incubator/NiFiProposal
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> Joe
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> = NiFi Proposal =
>>>>>>>>>>
>>>>>>>>>> == Abstract ==
>>>>>>>>>> NiFi is a dataflow system based on the concepts of flow-based
>>>>>>>>>>
>>>>>>>>>>  programming.
>>>>>>>>> == Proposal ==
>>>>>>>>>
>>>>>>>>>> NiFi supports powerful and scalable directed graphs of data
>>>>>>>>>> routing,
>>>>>>>>>> transformation, and system mediation logic.  Some of the
>>>>>>>>>> high-level
>>>>>>>>>> capabilities and objectives of NiFi include:
>>>>>>>>>>    * Web-based user interface for seamless experience between
>>>>>>>>>> design,
>>>>>>>>>> control, feedback, and monitoring of data flows
>>>>>>>>>>    * Highly configurable along several dimensions of quality of
>>>>>>>>>>
>>>>>>>>>>  service
>>>>>>>>>
>>>>>>>> such as loss tolerant versus guaranteed delivery, low latency versus
>>>>>>>
>>>>>>>> high throughput, and priority based queuing
>>>>>>>>>>    * Fine-grained data provenance for all data received, forked,
>>>>>>>>>> joined, cloned, modified, sent, and ultimately dropped as data
>>>>>>>>>>
>>>>>>>>>>  reaches
>>>>>>>>>
>>>>>>>> its configured end-state
>>>>>>>
>>>>>>>>    * Component-based extension model along well defined interfaces
>>>>>>>>>> enabling rapid development and effective testing
>>>>>>>>>>
>>>>>>>>>> == Background ==
>>>>>>>>>> Reliable and effective dataflow between systems can be difficult
>>>>>>>>>> whether you're running scripts on a laptop or have a massive
>>>>>>>>>> distributed computing system operated by numerous teams and
>>>>>>>>>> organizations.  As the volume and rate of data grows and as the
>>>>>>>>>>
>>>>>>>>>>  number
>>>>>>>>>
>>>>>>>> of systems, protocols, and formats increase and evolve so too does
>>>>>>>
>>>>>>>> the
>>>>>>>>>
>>>>>>>> complexity and need for greater insight and agility.  These are the
>>>>>>>
>>>>>>>> dataflow challenges that NiFi was built to tackle.
>>>>>>>>>>
>>>>>>>>>> NiFi is designed in a manner consistent with the core concepts
>>>>>>>>>> described in flow-based programming as originally documented by J.
>>>>>>>>>> Paul Morrison in the 1970s.  This model lends itself well to
>>>>>>>>>> visual
>>>>>>>>>> diagramming, concurrency, componentization, testing, and reuse.
>>>>>>>>>> In
>>>>>>>>>> addition to staying close to the fundamentals of flow-based
>>>>>>>>>> programming, NiFi provides integration system specific features
>>>>>>>>>> such
>>>>>>>>>> as: guaranteed delivery; back pressure; ability to gracefully
>>>>>>>>>> handle
>>>>>>>>>> backlogs and data surges; and an operator interface that enables
>>>>>>>>>> on-the-fly data flow generation, modification, and observation.
>>>>>>>>>>
>>>>>>>>>> == Rationale ==
>>>>>>>>>> NiFi provides a reliable, scalable, manageable and accountable
>>>>>>>>>> platform for developers and technical staff to create and evolve
>>>>>>>>>> powerful data flows.  Such a system is useful in many contexts
>>>>>>>>>> including large-scale enterprise integration, interaction with
>>>>>>>>>> cloud
>>>>>>>>>> services and frameworks, business to business, intra-departmental,
>>>>>>>>>>
>>>>>>>>>>  and
>>>>>>>>>
>>>>>>>> inter-departmental flows.  NiFi fits well within the Apache Software
>>>>>>>
>>>>>>>> Foundation (ASF) family as it depends on numerous ASF projects and
>>>>>>>>>> integrates with several others.  We also anticipate developing
>>>>>>>>>> extensions for several other ASF projects such as Cassandra,
>>>>>>>>>> Kafka,
>>>>>>>>>> and Storm in the near future.
>>>>>>>>>>
>>>>>>>>>> == Initial Goals ==
>>>>>>>>>>    * Ensure all dependencies are compliant with Apache License
>>>>>>>>>>
>>>>>>>>>>  version
>>>>>>>>>
>>>>>>>> 2.0 and all that all code and documentation artifacts have the
>>>>>>>
>>>>>>>> correct
>>>>>>>>>
>>>>>>>> Apache licensing markings and notice.
>>>>>>>
>>>>>>>>    * Establish a formal release process and schedule, allowing for
>>>>>>>>>> dependable release cycles in a manner consistent with the Apache
>>>>>>>>>> development process.
>>>>>>>>>>    * Determine and establish a mechanism, possibly including a
>>>>>>>>>> sub-project construct, that allows for extensions to the core
>>>>>>>>>> application to occur at a pace that differs from the core
>>>>>>>>>>
>>>>>>>>>>  application
>>>>>>>>>
>>>>>>>> itself.
>>>>>>>
>>>>>>>> == Current Status ==
>>>>>>>>>> === Meritocracy ===
>>>>>>>>>> An integration platform is only as good as its ability to
>>>>>>>>>> integrate
>>>>>>>>>> systems in a reliable, timely, and repeatable manner.  The same
>>>>>>>>>> can
>>>>>>>>>>
>>>>>>>>>>  be
>>>>>>>>>
>>>>>>>> said of its ability to attract talent and a variety of perspectives
>>>>>>>
>>>>>>>> as
>>>>>>>>>
>>>>>>>> integration systems by their nature are always evolving.  We will
>>>>>>>
>>>>>>>> actively seek help and encourage promotion of influence in the
>>>>>>>>>>
>>>>>>>>>>  project
>>>>>>>>>
>>>>>>>> through meritocracy.
>>>>>>>
>>>>>>>> === Community ===
>>>>>>>>>> Over the past several years, NiFi has developed a strong community
>>>>>>>>>>
>>>>>>>>>>  of
>>>>>>>>>
>>>>>>>> both developers and operators within the U.S. government.  We look
>>>>>>>
>>>>>>>> forward to helping grow this to a broader base of industries.
>>>>>>>>>>
>>>>>>>>>> === Core Developers ===
>>>>>>>>>> The initial core developers are employed by the National Security
>>>>>>>>>> Agency and defense contractors.  We will work to grow the
>>>>>>>>>> community
>>>>>>>>>> among a more diverse set of developers and industries.
>>>>>>>>>>
>>>>>>>>>> === Alignment ===
>>>>>>>>>>
>>>>>>>>>>  From its inception, NiFi was developed with an open source
>>>>>>>>>>
>>>>>>>>> philosophy
>>>>>>>>
>>>>>>> in mind and with the hopes of eventually being truly open sourced.
>>>>>>>
>>>>>>>> The Apache way is consistent with the approach we have taken to
>>>>>>>>>>
>>>>>>>>>>  date.
>>>>>>>>>
>>>>>>>> The ASF clearly provides a mature and effective environment for
>>>>>>>
>>>>>>>> successful development as is evident across the spectrum of
>>>>>>>>>>
>>>>>>>>>>  well-known
>>>>>>>>>
>>>>>>>> projects.  Further, NiFi depends on numerous ASF libraries and
>>>>>>>
>>>>>>>> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
>>>>>>>>>> HttpClient, Jakarta and Maven.  We also anticipate extensions and
>>>>>>>>>> dependencies with several more ASF projects, including Accumulo,
>>>>>>>>>>
>>>>>>>>>>  Avro,
>>>>>>>>>
>>>>>>>> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
>>>>>>>
>>>>>>>> This existing alignment with Apache and the desired community makes
>>>>>>>>>> the Apache Incubator a good fit for NiFi.
>>>>>>>>>>
>>>>>>>>>> == Known Risks ==
>>>>>>>>>> === Orphaned Products ===
>>>>>>>>>> Risk of orphaning is minimal.  The project user and developer base
>>>>>>>>>>
>>>>>>>>>>  is
>>>>>>>>>
>>>>>>>> substantial, growing, and there is already extensive operational use
>>>>>>>
>>>>>>>> of NiFi.
>>>>>>>>>>
>>>>>>>>>> === Inexperience with Open Source ===
>>>>>>>>>> The initial committers to NiFi have limited experience with true
>>>>>>>>>>
>>>>>>>>>>  open
>>>>>>>>>
>>>>>>>> source software development.  However, despite the project origins
>>>>>>>
>>>>>>>> being from closed source development we have modelled our behavior
>>>>>>>>>>
>>>>>>>>>>  and
>>>>>>>>>
>>>>>>>> community development on The Apache Way to the greatest extent
>>>>>>>
>>>>>>>> possible.  This environment includes widely accessible source code
>>>>>>>>>> repositories, published artifacts, ticket tracking, and extensive
>>>>>>>>>> documentation. We also encourage contributions and frequent debate
>>>>>>>>>>
>>>>>>>>>>  and
>>>>>>>>>
>>>>>>>> hold regular, collaborative discussions through e-mail, chat rooms,
>>>>>>>
>>>>>>>> and in-person meet-ups.  We are committed to the ideals of open
>>>>>>>>>>
>>>>>>>>>>  source
>>>>>>>>>
>>>>>>>> software and will eagerly seek out mentors and sponsors who can help
>>>>>>>
>>>>>>>> us quickly come up to speed.
>>>>>>>>>>
>>>>>>>>>> === Homogenous Developers ===
>>>>>>>>>> The initial committers of NiFi come from a limited set of entities
>>>>>>>>>> though we are committed to recruiting and developing additional
>>>>>>>>>> committers from a broad spectrum of industries and backgrounds.
>>>>>>>>>>
>>>>>>>>>> === Reliance on Salaried Developers ===
>>>>>>>>>> We expect NiFi development to continue on salaried time and
>>>>>>>>>> through
>>>>>>>>>> volunteer time.  The initial committers are paid by their
>>>>>>>>>> employers
>>>>>>>>>>
>>>>>>>>>>  to
>>>>>>>>>
>>>>>>>> contribute to this project.  We are committed to developing and
>>>>>>>
>>>>>>>> recruiting participation from developers both salaried and
>>>>>>>>>> non-salaried.
>>>>>>>>>>
>>>>>>>>>> === Relationship with other Apache Projects ===
>>>>>>>>>> As described in the alignment section, NiFi is already heavily
>>>>>>>>>> dependent on other ASF projects and we anticipate further
>>>>>>>>>> dependence
>>>>>>>>>> and integration with new and emerging projects in the Apache
>>>>>>>>>> family.
>>>>>>>>>>
>>>>>>>>>> === An Excessive Fascination with the Apache Brand ===
>>>>>>>>>> We respect the laudable Apache brand and that is certainly a
>>>>>>>>>> factor
>>>>>>>>>>
>>>>>>>>>>  in
>>>>>>>>>
>>>>>>>> the decision to propose NiFi for the Apache Incubator.  We intend to
>>>>>>>
>>>>>>>> provide a great deal of energy and capability to the ASF through
>>>>>>>>>>
>>>>>>>>>>  this
>>>>>>>>>
>>>>>>>> project.  We will be sensitive to and respectful of any overuse of
>>>>>>>
>>>>>>>> the
>>>>>>>>>
>>>>>>>> Apache brand and ensure our focus remains on how we benefit the
>>>>>>>
>>>>>>>> Apache
>>>>>>>>>
>>>>>>>> community.
>>>>>>>
>>>>>>>> === Documentation ===
>>>>>>>>>> At this time there is no NiFi documentation on the web.  However,
>>>>>>>>>> we
>>>>>>>>>> have extensive documentation included within the application that
>>>>>>>>>> details usage of the many functions.  We will be rapidly expanding
>>>>>>>>>>
>>>>>>>>>>  the
>>>>>>>>>
>>>>>>>> available documentation to cover things like installation, developer
>>>>>>>
>>>>>>>> guide, frequently asked questions, best practices, and more.  This
>>>>>>>>>> documentation will be posted to the NiFi wiki at apache.org.
>>>>>>>>>>
>>>>>>>>>> == Initial Source ==
>>>>>>>>>> NiFi has been in active development since late 2006 with
>>>>>>>>>>
>>>>>>>>>>  contributions
>>>>>>>>>
>>>>>>>> >from dozens of developers and feedback from hundreds of users and
>>>>>>>
>>>>>>>> developers.  The core codebase is written in Java and includes
>>>>>>>>>> detailed Javadocs and feature documentation.
>>>>>>>>>>
>>>>>>>>>> == Source and Intellectual Property Submission ==
>>>>>>>>>> Previously referred to as Niagarafiles, the NiFi code and
>>>>>>>>>> documentation materials will be submitted by the National Security
>>>>>>>>>> Agency.  NiFi has been developed by a mix of government employees
>>>>>>>>>>
>>>>>>>>>>  and
>>>>>>>>>
>>>>>>>> private companies under government contract.  Material developed by
>>>>>>>
>>>>>>>> the government employees is in the public domain and no U.S.
>>>>>>>>>>
>>>>>>>>>>  copyright
>>>>>>>>>
>>>>>>>> exists in works of the federal government.  For the contractor
>>>>>>>
>>>>>>>> developed material in the initial submission, the U.S. Government
>>>>>>>>>>
>>>>>>>>>>  has
>>>>>>>>>
>>>>>>>> sufficient authority to open source per DFARS 252.227-7014.  NSA
>>>>>>>
>>>>>>>> has
>>>>>>>>>
>>>>>>>> submitted the Software Grant Agreement and Corporate Contributor
>>>>>>>
>>>>>>>> License Agreement to the Apache Software Foundation.
>>>>>>>>>>
>>>>>>>>>> == External Dependencies ==
>>>>>>>>>> We have at least one dependency on an LGPL library which we will
>>>>>>>>>> promptly address.  Otherwise, we believe all current dependencies
>>>>>>>>>>
>>>>>>>>>>  are
>>>>>>>>>
>>>>>>>> compatible with the ASF guidelines.  Our dependency licenses come
>>>>>>>
>>>>>>>> from
>>>>>>>>>
>>>>>>>> the following license styles:  Apache v 2.0, BSD, Public Domain,
>>>>>>>
>>>>>>>> Eclipse Public v1, MIT, CDDL v1.
>>>>>>>>>>
>>>>>>>>>> == Cryptography ==
>>>>>>>>>> Consistent with http://www.apache.org/licenses/exports/ we
>>>>>>>>>> believe
>>>>>>>>>> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
>>>>>>>>>> cryptographic algorithms but is designed to use algorithms
>>>>>>>>>> provided
>>>>>>>>>>
>>>>>>>>>>  by
>>>>>>>>>
>>>>>>>> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
>>>>>>>
>>>>>>>> These cryptographic algorithm providers are used to support SSL,
>>>>>>>>>> SSH/SFTP, and the encryption and decryption of sensitive
>>>>>>>>>> properties.
>>>>>>>>>> In the event that it becomes necessary we will engage with
>>>>>>>>>>
>>>>>>>>>>  appropriate
>>>>>>>>>
>>>>>>>> Apache members to ensure we file any necessary paperwork or
>>>>>>>
>>>>>>>> clarified
>>>>>>>>>
>>>>>>>> any cryptographic export license concerns.
>>>>>>>
>>>>>>>> == Required Resources ==
>>>>>>>>>> === Mailing Lists ===
>>>>>>>>>>    * user@nifi.incubator.apache.org
>>>>>>>>>>    * dev@nifi.incubator.apache.org
>>>>>>>>>>    * private@nifi.incubator.apache.org
>>>>>>>>>>    * commits@nifi.incubator.apache.org
>>>>>>>>>>
>>>>>>>>>> === Source Control ===
>>>>>>>>>> NiFi requests use of Git for source control
>>>>>>>>>> (git://git.apache.org/nifi.git).  We request a writeable Git repo
>>>>>>>>>>
>>>>>>>>>>  for
>>>>>>>>>
>>>>>>>> NiFi with mirroring to be setup to Github through INFRA.  We request
>>>>>>>
>>>>>>>> sponsor Benson Margulies (bimargulies) to assist with creating the
>>>>>>>>>> INFRA ticket for this.
>>>>>>>>>>
>>>>>>>>>> === Issue Tracking ===
>>>>>>>>>> JIRA NiFi (NIFI)
>>>>>>>>>>
>>>>>>>>>> === Initial Committers ===
>>>>>>>>>>    * Brandon Devries<brandon.devries at gmail dot com>
>>>>>>>>>>    * Matt Gilman<matt.c.gilman at gmail dot com>
>>>>>>>>>>    * Tony Kurc<trkurc at gmail dot com>
>>>>>>>>>>    * Mark Payne<markap14 at hotmail dot com>
>>>>>>>>>>    * Adam Taft<adam at adamtaft dot com>
>>>>>>>>>>    * Joseph Witt<joewitt at gmail dot com>
>>>>>>>>>>
>>>>>>>>>> === Affiliations ===
>>>>>>>>>>    * Brandon Devries (Requitest, Inc.)
>>>>>>>>>>    * Matt Gilman (Raytheon)
>>>>>>>>>>    * Tony Kurc (National Security Agency)
>>>>>>>>>>    * Mark Payne (Sotera Defense Solutions, Inc.)
>>>>>>>>>>    * Adam Taft (Requitest, Inc.)
>>>>>>>>>>    * Joseph Witt (National Security Agency)
>>>>>>>>>>
>>>>>>>>>> == Sponsors ==
>>>>>>>>>> === Champion ===
>>>>>>>>>>    * Benson Margulies (Basis Technology)<bimargulies at apache dot
>>>>>>>>>>
>>>>>>>>>>  org>
>>>>>>>>>
>>>>>>>> === Nominated Mentors ===
>>>>>>>>
>>>>>>>>>    * Drew Farris (Booz Allen Hamilton)<drew at apache dot org>
>>>>>>>>>>    * Brock Noland (Cloudera)<brock at apache dot org>
>>>>>>>>>>    * Billie Rinaldi (Hortonworks)<billie at apache dot org>
>>>>>>>>>>
>>>>>>>>>> === Sponsoring Entity ===
>>>>>>>>>> We request the Apache Incubator to sponsor this project.
>>>>>>>>>>
>>>>>>>>>>  ------------------------------------------------------------
>>>>>>>>> ---------
>>>>>>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>>>>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  --
>>>>>>> Sean
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>  ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>
>>>>
>>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
>

Re: [PROPOSAL] NiFi for Incubation

[Joe Witt <jo...@gmail.com>]

Andrew - appreciate your comments and offer to assist as a mentor.  Now
that the vote for incubation has kicked off i'm not sure what i'm allowed
to do or not.  Will check will Benson.

For what it is worth I agree Sean raised a great point.  I think it will be
an important part of our community building effort.  And to echo Josh's
point this proposal represents not just our intent to grow the community
outside our current home but rather move homes - to the ASF.  We're very
excited to have this opportunity.

Thanks
Joe

On Fri, Nov 21, 2014 at 5:19 PM, Josh Elser <el...@apache.org> wrote:

> Thanks for the clarification -- that wasn't how it came across to me.
>
>
> Andrew Purtell wrote:
>
>> I didn't make an assertion *about* Accumulo, what I said exactly was:
>>
>>          "that Accumulo may or may not suffer from "bifurcation" is
>> debatable"
>>
>> I assert there are two sides to that argument (I've heard them both)​​,
>> merely to suggest that Sean's concerns should not be dismissed as they
>> were.
>>
>>
>> On Fri, Nov 21, 2014 at 2:11 PM, Josh Elser<el...@apache.org>  wrote:
>>
>>  I would tend to disagree with your assertion about Accumulo, but this is
>>> neither the time nor the place to discuss Accumulo's strengths and
>>> weaknesses.
>>>
>>> I'll make the assumption that request to enter incubation is a strong
>>> intent to grow outside of the project's current home.
>>>
>>> +1 (non-binding) for entry
>>>
>>>
>>> Andrew Purtell wrote:
>>>
>>>     From what I understand, that Accumulo may or may not suffer from
>>>> "bifurcation" is debatable despite this blanket claim otherwise.
>>>>
>>>> Regardless, Sean's point is valid, even if only a theoretical concern
>>>> for
>>>> NiFi at this point. If the podling is interested in growing outside
>>>> community I'd be happy to sign on as a mentor to help that happen.
>>>>
>>>> On Thu, Nov 20, 2014 at 3:32 AM, Benson Margulies<bimargulies@gmail.com
>>>> >
>>>> wrote:
>>>>
>>>>   Sean,
>>>>
>>>>> The precedent of Accumulo is that the govt people and agencies involved
>>>>> are
>>>>> ready and able to have their staff collaborate openly in an Apache
>>>>> community. There's no need to contemplate bifurcation; we have this
>>>>> proposal because the management recognizes that this collaboration
>>>>> produces
>>>>> better stuff that solves more problems than the 'inside the tent'
>>>>> alternative.
>>>>>
>>>>> --benson
>>>>>
>>>>>
>>>>> On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey<bu...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>   I'm really excited to see NiFi come to the incubator; it'd be a great
>>>>>
>>>>>> addition to the ASF.
>>>>>>
>>>>>> A few points in the proposal:
>>>>>>
>>>>>>   == Initial Goals ==
>>>>>>
>>>>>>> One of these should be to grow the community outside of the current
>>>>>>>
>>>>>> niche,
>>>>>
>>>>>  IMHO.
>>>>>>
>>>>>> More on this below under "orphaned projects"
>>>>>>
>>>>>>     * Determine and establish a mechanism, possibly including a
>>>>>>
>>>>>>> sub-project construct, that allows for extensions to the core
>>>>>>>> application to occur at a pace that differs from the core
>>>>>>>> application
>>>>>>>> itself.
>>>>>>>>
>>>>>>>>  I don't think the proposal needs to include the "e.g. with
>>>>>>>
>>>>>> sub-projects"
>>>>>> part. Just noting
>>>>>> that your goals in the incubator are to address the need to have
>>>>>>
>>>>>>  different
>>>>>
>>>>>  release cycles
>>>>>> for core and extensions is sufficient.
>>>>>>
>>>>>>
>>>>>>   === Community ===
>>>>>>
>>>>>>> Over the past several years, NiFi has developed a strong community of
>>>>>>>> both developers and operators within the U.S. government.  We look
>>>>>>>> forward to helping grow this to a broader base of industries.
>>>>>>>>
>>>>>>>>   How much, if any, of this community do you expect to engage via
>>>>>>>> the
>>>>>>>>
>>>>>>> customary project
>>>>>> lists once NiFi is established within the ASF? Will the project be
>>>>>> able
>>>>>>
>>>>>>  to
>>>>>
>>>>>  leverage this
>>>>>> established group?
>>>>>>
>>>>>>
>>>>>>   === Orphaned Products ===
>>>>>>
>>>>>>> Risk of orphaning is minimal.  The project user and developer base is
>>>>>>>> substantial, growing, and there is already extensive operational use
>>>>>>>> of NiFi.
>>>>>>>>
>>>>>>>>  Given that the established base is internal to the U.S.
>>>>>>> government, I'd
>>>>>>>
>>>>>> encourage the
>>>>>> podling to consider the risk of a bifurcated project should a
>>>>>> substantial
>>>>>> outside
>>>>>> community fail to emerge or if those internal users should fail to
>>>>>> engage
>>>>>> with the
>>>>>> outside community.
>>>>>>
>>>>>> You cover a related issue in your "Homogenous Developers" section.
>>>>>> But I
>>>>>> think
>>>>>> building on the "Community" section of the current state to call this
>>>>>> out
>>>>>> as an
>>>>>> independent issue is worthwhile.
>>>>>>
>>>>>>
>>>>>>   possible.  This environment includes widely accessible source code
>>>>>>
>>>>>>> repositories, published artifacts, ticket tracking, and extensive
>>>>>>>> documentation. We also encourage contributions and frequent debate
>>>>>>>> and
>>>>>>>> hold regular, collaborative discussions through e-mail, chat rooms,
>>>>>>>> and in-person meet-ups.
>>>>>>>>
>>>>>>>>  Do you anticipate any difficulties moving these established
>>>>>>>
>>>>>> communication
>>>>>> mechanisms to ASF public lists?
>>>>>>
>>>>>>   === Documentation ===
>>>>>>
>>>>>>> At this time there is no NiFi documentation on the web.  However, we
>>>>>>>> have extensive documentation included within the application that
>>>>>>>> details usage of the many functions.  We will be rapidly expanding
>>>>>>>> the
>>>>>>>> available documentation to cover things like installation, developer
>>>>>>>> guide, frequently asked questions, best practices, and more.  This
>>>>>>>> documentation will be posted to the NiFi wiki at apache.org.
>>>>>>>>
>>>>>>>>  I love projects that start with documentation. :)
>>>>>>>
>>>>>> I don't think the proposal needs to include that the documentation
>>>>>> will
>>>>>>
>>>>>>  be
>>>>>
>>>>>  posted
>>>>>> to the NiFi wiki, since that's an implementation detail. Just say
>>>>>> "this
>>>>>> documentation
>>>>>> will be made available via the NiFi project's use of incubator infra."
>>>>>>
>>>>>> (I'll save detail for the eventual dev@ list, but you should strongly
>>>>>> consider not
>>>>>> using the wiki to host this documentation.)
>>>>>>
>>>>>> -Sean
>>>>>>
>>>>>> On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland<br...@cloudera.com>
>>>>>>
>>>>>>  wrote:
>>>>>
>>>>>  Hi Joe,
>>>>>>
>>>>>>> I know you've done a tremendous amount of work to make this happen
>>>>>>> so I
>>>>>>>
>>>>>>>  am
>>>>>>
>>>>>>  extremely happy this is *finally* making it's way to the incubator!
>>>>>>>
>>>>>>> I look forward to helping in anyway I can.
>>>>>>>
>>>>>>> Cheers!
>>>>>>> Brock
>>>>>>>
>>>>>>> On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980)<
>>>>>>> chris.a.mattmann@jpl.nasa.gov>   wrote:
>>>>>>>
>>>>>>>   This is *fan freakin¹ tastic* Sounds like an awesome  project and
>>>>>>>
>>>>>>>> glad to hear a relationship to Tika! Awesome to see more government
>>>>>>>> projects coming into the ASF!
>>>>>>>>
>>>>>>>> you already have a great set of mentors and I don¹t really have more
>>>>>>>> time on my plate, but really happy and will try and monitor and help
>>>>>>>> on the lists.
>>>>>>>>
>>>>>>>> Cheers!
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> Chris Mattmann, Ph.D.
>>>>>>>> Chief Architect
>>>>>>>> Instrument Software and Science Data Systems Section (398)
>>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>> Office: 168-519, Mailstop: 168-527
>>>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>>>> WWW:  http://sunset.usc.edu/~mattmann/
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> Adjunct Associate Professor, Computer Science Department
>>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Joe Witt<jo...@gmail.com>
>>>>>>>> Reply-To: "general@incubator.apache.org"<
>>>>>>>>
>>>>>>>>  general@incubator.apache.org
>>>>>>>
>>>>>> Date: Thursday, November 20, 2014 at 3:02 AM
>>>>>>
>>>>>>> To: "general@incubator.apache.org"<ge...@incubator.apache.org>
>>>>>>>> Subject: [PROPOSAL] NiFi for Incubation
>>>>>>>>
>>>>>>>>   Hello,
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I would like to propose NiFi as an Apache Incubator Project.
>>>>>>>>>
>>>>>>>>> In addition to the copy provided below the Wiki version of the
>>>>>>>>> proposal can be found here:
>>>>>>>>> http://wiki.apache.org/incubator/NiFiProposal
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Joe
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> = NiFi Proposal =
>>>>>>>>>
>>>>>>>>> == Abstract ==
>>>>>>>>> NiFi is a dataflow system based on the concepts of flow-based
>>>>>>>>>
>>>>>>>>>  programming.
>>>>>>>> == Proposal ==
>>>>>>>>
>>>>>>>>> NiFi supports powerful and scalable directed graphs of data
>>>>>>>>> routing,
>>>>>>>>> transformation, and system mediation logic.  Some of the high-level
>>>>>>>>> capabilities and objectives of NiFi include:
>>>>>>>>>    * Web-based user interface for seamless experience between
>>>>>>>>> design,
>>>>>>>>> control, feedback, and monitoring of data flows
>>>>>>>>>    * Highly configurable along several dimensions of quality of
>>>>>>>>>
>>>>>>>>>  service
>>>>>>>>
>>>>>>> such as loss tolerant versus guaranteed delivery, low latency versus
>>>>>>
>>>>>>> high throughput, and priority based queuing
>>>>>>>>>    * Fine-grained data provenance for all data received, forked,
>>>>>>>>> joined, cloned, modified, sent, and ultimately dropped as data
>>>>>>>>>
>>>>>>>>>  reaches
>>>>>>>>
>>>>>>> its configured end-state
>>>>>>
>>>>>>>    * Component-based extension model along well defined interfaces
>>>>>>>>> enabling rapid development and effective testing
>>>>>>>>>
>>>>>>>>> == Background ==
>>>>>>>>> Reliable and effective dataflow between systems can be difficult
>>>>>>>>> whether you're running scripts on a laptop or have a massive
>>>>>>>>> distributed computing system operated by numerous teams and
>>>>>>>>> organizations.  As the volume and rate of data grows and as the
>>>>>>>>>
>>>>>>>>>  number
>>>>>>>>
>>>>>>> of systems, protocols, and formats increase and evolve so too does
>>>>>>
>>>>>>> the
>>>>>>>>
>>>>>>> complexity and need for greater insight and agility.  These are the
>>>>>>
>>>>>>> dataflow challenges that NiFi was built to tackle.
>>>>>>>>>
>>>>>>>>> NiFi is designed in a manner consistent with the core concepts
>>>>>>>>> described in flow-based programming as originally documented by J.
>>>>>>>>> Paul Morrison in the 1970s.  This model lends itself well to visual
>>>>>>>>> diagramming, concurrency, componentization, testing, and reuse.  In
>>>>>>>>> addition to staying close to the fundamentals of flow-based
>>>>>>>>> programming, NiFi provides integration system specific features
>>>>>>>>> such
>>>>>>>>> as: guaranteed delivery; back pressure; ability to gracefully
>>>>>>>>> handle
>>>>>>>>> backlogs and data surges; and an operator interface that enables
>>>>>>>>> on-the-fly data flow generation, modification, and observation.
>>>>>>>>>
>>>>>>>>> == Rationale ==
>>>>>>>>> NiFi provides a reliable, scalable, manageable and accountable
>>>>>>>>> platform for developers and technical staff to create and evolve
>>>>>>>>> powerful data flows.  Such a system is useful in many contexts
>>>>>>>>> including large-scale enterprise integration, interaction with
>>>>>>>>> cloud
>>>>>>>>> services and frameworks, business to business, intra-departmental,
>>>>>>>>>
>>>>>>>>>  and
>>>>>>>>
>>>>>>> inter-departmental flows.  NiFi fits well within the Apache Software
>>>>>>
>>>>>>> Foundation (ASF) family as it depends on numerous ASF projects and
>>>>>>>>> integrates with several others.  We also anticipate developing
>>>>>>>>> extensions for several other ASF projects such as Cassandra, Kafka,
>>>>>>>>> and Storm in the near future.
>>>>>>>>>
>>>>>>>>> == Initial Goals ==
>>>>>>>>>    * Ensure all dependencies are compliant with Apache License
>>>>>>>>>
>>>>>>>>>  version
>>>>>>>>
>>>>>>> 2.0 and all that all code and documentation artifacts have the
>>>>>>
>>>>>>> correct
>>>>>>>>
>>>>>>> Apache licensing markings and notice.
>>>>>>
>>>>>>>    * Establish a formal release process and schedule, allowing for
>>>>>>>>> dependable release cycles in a manner consistent with the Apache
>>>>>>>>> development process.
>>>>>>>>>    * Determine and establish a mechanism, possibly including a
>>>>>>>>> sub-project construct, that allows for extensions to the core
>>>>>>>>> application to occur at a pace that differs from the core
>>>>>>>>>
>>>>>>>>>  application
>>>>>>>>
>>>>>>> itself.
>>>>>>
>>>>>>> == Current Status ==
>>>>>>>>> === Meritocracy ===
>>>>>>>>> An integration platform is only as good as its ability to integrate
>>>>>>>>> systems in a reliable, timely, and repeatable manner.  The same can
>>>>>>>>>
>>>>>>>>>  be
>>>>>>>>
>>>>>>> said of its ability to attract talent and a variety of perspectives
>>>>>>
>>>>>>> as
>>>>>>>>
>>>>>>> integration systems by their nature are always evolving.  We will
>>>>>>
>>>>>>> actively seek help and encourage promotion of influence in the
>>>>>>>>>
>>>>>>>>>  project
>>>>>>>>
>>>>>>> through meritocracy.
>>>>>>
>>>>>>> === Community ===
>>>>>>>>> Over the past several years, NiFi has developed a strong community
>>>>>>>>>
>>>>>>>>>  of
>>>>>>>>
>>>>>>> both developers and operators within the U.S. government.  We look
>>>>>>
>>>>>>> forward to helping grow this to a broader base of industries.
>>>>>>>>>
>>>>>>>>> === Core Developers ===
>>>>>>>>> The initial core developers are employed by the National Security
>>>>>>>>> Agency and defense contractors.  We will work to grow the community
>>>>>>>>> among a more diverse set of developers and industries.
>>>>>>>>>
>>>>>>>>> === Alignment ===
>>>>>>>>>
>>>>>>>>>  From its inception, NiFi was developed with an open source
>>>>>>>>>
>>>>>>>> philosophy
>>>>>>>
>>>>>> in mind and with the hopes of eventually being truly open sourced.
>>>>>>
>>>>>>> The Apache way is consistent with the approach we have taken to
>>>>>>>>>
>>>>>>>>>  date.
>>>>>>>>
>>>>>>> The ASF clearly provides a mature and effective environment for
>>>>>>
>>>>>>> successful development as is evident across the spectrum of
>>>>>>>>>
>>>>>>>>>  well-known
>>>>>>>>
>>>>>>> projects.  Further, NiFi depends on numerous ASF libraries and
>>>>>>
>>>>>>> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
>>>>>>>>> HttpClient, Jakarta and Maven.  We also anticipate extensions and
>>>>>>>>> dependencies with several more ASF projects, including Accumulo,
>>>>>>>>>
>>>>>>>>>  Avro,
>>>>>>>>
>>>>>>> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
>>>>>>
>>>>>>> This existing alignment with Apache and the desired community makes
>>>>>>>>> the Apache Incubator a good fit for NiFi.
>>>>>>>>>
>>>>>>>>> == Known Risks ==
>>>>>>>>> === Orphaned Products ===
>>>>>>>>> Risk of orphaning is minimal.  The project user and developer base
>>>>>>>>>
>>>>>>>>>  is
>>>>>>>>
>>>>>>> substantial, growing, and there is already extensive operational use
>>>>>>
>>>>>>> of NiFi.
>>>>>>>>>
>>>>>>>>> === Inexperience with Open Source ===
>>>>>>>>> The initial committers to NiFi have limited experience with true
>>>>>>>>>
>>>>>>>>>  open
>>>>>>>>
>>>>>>> source software development.  However, despite the project origins
>>>>>>
>>>>>>> being from closed source development we have modelled our behavior
>>>>>>>>>
>>>>>>>>>  and
>>>>>>>>
>>>>>>> community development on The Apache Way to the greatest extent
>>>>>>
>>>>>>> possible.  This environment includes widely accessible source code
>>>>>>>>> repositories, published artifacts, ticket tracking, and extensive
>>>>>>>>> documentation. We also encourage contributions and frequent debate
>>>>>>>>>
>>>>>>>>>  and
>>>>>>>>
>>>>>>> hold regular, collaborative discussions through e-mail, chat rooms,
>>>>>>
>>>>>>> and in-person meet-ups.  We are committed to the ideals of open
>>>>>>>>>
>>>>>>>>>  source
>>>>>>>>
>>>>>>> software and will eagerly seek out mentors and sponsors who can help
>>>>>>
>>>>>>> us quickly come up to speed.
>>>>>>>>>
>>>>>>>>> === Homogenous Developers ===
>>>>>>>>> The initial committers of NiFi come from a limited set of entities
>>>>>>>>> though we are committed to recruiting and developing additional
>>>>>>>>> committers from a broad spectrum of industries and backgrounds.
>>>>>>>>>
>>>>>>>>> === Reliance on Salaried Developers ===
>>>>>>>>> We expect NiFi development to continue on salaried time and through
>>>>>>>>> volunteer time.  The initial committers are paid by their employers
>>>>>>>>>
>>>>>>>>>  to
>>>>>>>>
>>>>>>> contribute to this project.  We are committed to developing and
>>>>>>
>>>>>>> recruiting participation from developers both salaried and
>>>>>>>>> non-salaried.
>>>>>>>>>
>>>>>>>>> === Relationship with other Apache Projects ===
>>>>>>>>> As described in the alignment section, NiFi is already heavily
>>>>>>>>> dependent on other ASF projects and we anticipate further
>>>>>>>>> dependence
>>>>>>>>> and integration with new and emerging projects in the Apache
>>>>>>>>> family.
>>>>>>>>>
>>>>>>>>> === An Excessive Fascination with the Apache Brand ===
>>>>>>>>> We respect the laudable Apache brand and that is certainly a factor
>>>>>>>>>
>>>>>>>>>  in
>>>>>>>>
>>>>>>> the decision to propose NiFi for the Apache Incubator.  We intend to
>>>>>>
>>>>>>> provide a great deal of energy and capability to the ASF through
>>>>>>>>>
>>>>>>>>>  this
>>>>>>>>
>>>>>>> project.  We will be sensitive to and respectful of any overuse of
>>>>>>
>>>>>>> the
>>>>>>>>
>>>>>>> Apache brand and ensure our focus remains on how we benefit the
>>>>>>
>>>>>>> Apache
>>>>>>>>
>>>>>>> community.
>>>>>>
>>>>>>> === Documentation ===
>>>>>>>>> At this time there is no NiFi documentation on the web.  However,
>>>>>>>>> we
>>>>>>>>> have extensive documentation included within the application that
>>>>>>>>> details usage of the many functions.  We will be rapidly expanding
>>>>>>>>>
>>>>>>>>>  the
>>>>>>>>
>>>>>>> available documentation to cover things like installation, developer
>>>>>>
>>>>>>> guide, frequently asked questions, best practices, and more.  This
>>>>>>>>> documentation will be posted to the NiFi wiki at apache.org.
>>>>>>>>>
>>>>>>>>> == Initial Source ==
>>>>>>>>> NiFi has been in active development since late 2006 with
>>>>>>>>>
>>>>>>>>>  contributions
>>>>>>>>
>>>>>>> >from dozens of developers and feedback from hundreds of users and
>>>>>>
>>>>>>> developers.  The core codebase is written in Java and includes
>>>>>>>>> detailed Javadocs and feature documentation.
>>>>>>>>>
>>>>>>>>> == Source and Intellectual Property Submission ==
>>>>>>>>> Previously referred to as Niagarafiles, the NiFi code and
>>>>>>>>> documentation materials will be submitted by the National Security
>>>>>>>>> Agency.  NiFi has been developed by a mix of government employees
>>>>>>>>>
>>>>>>>>>  and
>>>>>>>>
>>>>>>> private companies under government contract.  Material developed by
>>>>>>
>>>>>>> the government employees is in the public domain and no U.S.
>>>>>>>>>
>>>>>>>>>  copyright
>>>>>>>>
>>>>>>> exists in works of the federal government.  For the contractor
>>>>>>
>>>>>>> developed material in the initial submission, the U.S. Government
>>>>>>>>>
>>>>>>>>>  has
>>>>>>>>
>>>>>>> sufficient authority to open source per DFARS 252.227-7014.  NSA
>>>>>>
>>>>>>> has
>>>>>>>>
>>>>>>> submitted the Software Grant Agreement and Corporate Contributor
>>>>>>
>>>>>>> License Agreement to the Apache Software Foundation.
>>>>>>>>>
>>>>>>>>> == External Dependencies ==
>>>>>>>>> We have at least one dependency on an LGPL library which we will
>>>>>>>>> promptly address.  Otherwise, we believe all current dependencies
>>>>>>>>>
>>>>>>>>>  are
>>>>>>>>
>>>>>>> compatible with the ASF guidelines.  Our dependency licenses come
>>>>>>
>>>>>>> from
>>>>>>>>
>>>>>>> the following license styles:  Apache v 2.0, BSD, Public Domain,
>>>>>>
>>>>>>> Eclipse Public v1, MIT, CDDL v1.
>>>>>>>>>
>>>>>>>>> == Cryptography ==
>>>>>>>>> Consistent with http://www.apache.org/licenses/exports/ we believe
>>>>>>>>> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
>>>>>>>>> cryptographic algorithms but is designed to use algorithms provided
>>>>>>>>>
>>>>>>>>>  by
>>>>>>>>
>>>>>>> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
>>>>>>
>>>>>>> These cryptographic algorithm providers are used to support SSL,
>>>>>>>>> SSH/SFTP, and the encryption and decryption of sensitive
>>>>>>>>> properties.
>>>>>>>>> In the event that it becomes necessary we will engage with
>>>>>>>>>
>>>>>>>>>  appropriate
>>>>>>>>
>>>>>>> Apache members to ensure we file any necessary paperwork or
>>>>>>
>>>>>>> clarified
>>>>>>>>
>>>>>>> any cryptographic export license concerns.
>>>>>>
>>>>>>> == Required Resources ==
>>>>>>>>> === Mailing Lists ===
>>>>>>>>>    * user@nifi.incubator.apache.org
>>>>>>>>>    * dev@nifi.incubator.apache.org
>>>>>>>>>    * private@nifi.incubator.apache.org
>>>>>>>>>    * commits@nifi.incubator.apache.org
>>>>>>>>>
>>>>>>>>> === Source Control ===
>>>>>>>>> NiFi requests use of Git for source control
>>>>>>>>> (git://git.apache.org/nifi.git).  We request a writeable Git repo
>>>>>>>>>
>>>>>>>>>  for
>>>>>>>>
>>>>>>> NiFi with mirroring to be setup to Github through INFRA.  We request
>>>>>>
>>>>>>> sponsor Benson Margulies (bimargulies) to assist with creating the
>>>>>>>>> INFRA ticket for this.
>>>>>>>>>
>>>>>>>>> === Issue Tracking ===
>>>>>>>>> JIRA NiFi (NIFI)
>>>>>>>>>
>>>>>>>>> === Initial Committers ===
>>>>>>>>>    * Brandon Devries<brandon.devries at gmail dot com>
>>>>>>>>>    * Matt Gilman<matt.c.gilman at gmail dot com>
>>>>>>>>>    * Tony Kurc<trkurc at gmail dot com>
>>>>>>>>>    * Mark Payne<markap14 at hotmail dot com>
>>>>>>>>>    * Adam Taft<adam at adamtaft dot com>
>>>>>>>>>    * Joseph Witt<joewitt at gmail dot com>
>>>>>>>>>
>>>>>>>>> === Affiliations ===
>>>>>>>>>    * Brandon Devries (Requitest, Inc.)
>>>>>>>>>    * Matt Gilman (Raytheon)
>>>>>>>>>    * Tony Kurc (National Security Agency)
>>>>>>>>>    * Mark Payne (Sotera Defense Solutions, Inc.)
>>>>>>>>>    * Adam Taft (Requitest, Inc.)
>>>>>>>>>    * Joseph Witt (National Security Agency)
>>>>>>>>>
>>>>>>>>> == Sponsors ==
>>>>>>>>> === Champion ===
>>>>>>>>>    * Benson Margulies (Basis Technology)<bimargulies at apache dot
>>>>>>>>>
>>>>>>>>>  org>
>>>>>>>>
>>>>>>> === Nominated Mentors ===
>>>>>>>
>>>>>>>>    * Drew Farris (Booz Allen Hamilton)<drew at apache dot org>
>>>>>>>>>    * Brock Noland (Cloudera)<brock at apache dot org>
>>>>>>>>>    * Billie Rinaldi (Hortonworks)<billie at apache dot org>
>>>>>>>>>
>>>>>>>>> === Sponsoring Entity ===
>>>>>>>>> We request the Apache Incubator to sponsor this project.
>>>>>>>>>
>>>>>>>>>  ------------------------------------------------------------
>>>>>>>> ---------
>>>>>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>>>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>  --
>>>>>> Sean
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>  ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>
>>>
>>>
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [PROPOSAL] NiFi for Incubation

[Josh Elser <el...@apache.org>]

Thanks for the clarification -- that wasn't how it came across to me.

Andrew Purtell wrote:
> I didn't make an assertion *about* Accumulo, what I said exactly was:
>
>          "that Accumulo may or may not suffer from "bifurcation" is
> debatable"
>
> I assert there are two sides to that argument (I've heard them both)​​,
> merely to suggest that Sean's concerns should not be dismissed as they
> were.
>
>
> On Fri, Nov 21, 2014 at 2:11 PM, Josh Elser<el...@apache.org>  wrote:
>
>> I would tend to disagree with your assertion about Accumulo, but this is
>> neither the time nor the place to discuss Accumulo's strengths and
>> weaknesses.
>>
>> I'll make the assumption that request to enter incubation is a strong
>> intent to grow outside of the project's current home.
>>
>> +1 (non-binding) for entry
>>
>>
>> Andrew Purtell wrote:
>>
>>>    From what I understand, that Accumulo may or may not suffer from
>>> "bifurcation" is debatable despite this blanket claim otherwise.
>>>
>>> Regardless, Sean's point is valid, even if only a theoretical concern for
>>> NiFi at this point. If the podling is interested in growing outside
>>> community I'd be happy to sign on as a mentor to help that happen.
>>>
>>> On Thu, Nov 20, 2014 at 3:32 AM, Benson Margulies<bi...@gmail.com>
>>> wrote:
>>>
>>>   Sean,
>>>> The precedent of Accumulo is that the govt people and agencies involved
>>>> are
>>>> ready and able to have their staff collaborate openly in an Apache
>>>> community. There's no need to contemplate bifurcation; we have this
>>>> proposal because the management recognizes that this collaboration
>>>> produces
>>>> better stuff that solves more problems than the 'inside the tent'
>>>> alternative.
>>>>
>>>> --benson
>>>>
>>>>
>>>> On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey<bu...@cloudera.com>
>>>> wrote:
>>>>
>>>>   I'm really excited to see NiFi come to the incubator; it'd be a great
>>>>> addition to the ASF.
>>>>>
>>>>> A few points in the proposal:
>>>>>
>>>>>   == Initial Goals ==
>>>>>> One of these should be to grow the community outside of the current
>>>> niche,
>>>>
>>>>> IMHO.
>>>>>
>>>>> More on this below under "orphaned projects"
>>>>>
>>>>>     * Determine and establish a mechanism, possibly including a
>>>>>>> sub-project construct, that allows for extensions to the core
>>>>>>> application to occur at a pace that differs from the core application
>>>>>>> itself.
>>>>>>>
>>>>>> I don't think the proposal needs to include the "e.g. with
>>>>> sub-projects"
>>>>> part. Just noting
>>>>> that your goals in the incubator are to address the need to have
>>>>>
>>>> different
>>>>
>>>>> release cycles
>>>>> for core and extensions is sufficient.
>>>>>
>>>>>
>>>>>   === Community ===
>>>>>>> Over the past several years, NiFi has developed a strong community of
>>>>>>> both developers and operators within the U.S. government.  We look
>>>>>>> forward to helping grow this to a broader base of industries.
>>>>>>>
>>>>>>>   How much, if any, of this community do you expect to engage via the
>>>>> customary project
>>>>> lists once NiFi is established within the ASF? Will the project be able
>>>>>
>>>> to
>>>>
>>>>> leverage this
>>>>> established group?
>>>>>
>>>>>
>>>>>   === Orphaned Products ===
>>>>>>> Risk of orphaning is minimal.  The project user and developer base is
>>>>>>> substantial, growing, and there is already extensive operational use
>>>>>>> of NiFi.
>>>>>>>
>>>>>> Given that the established base is internal to the U.S. government, I'd
>>>>> encourage the
>>>>> podling to consider the risk of a bifurcated project should a
>>>>> substantial
>>>>> outside
>>>>> community fail to emerge or if those internal users should fail to
>>>>> engage
>>>>> with the
>>>>> outside community.
>>>>>
>>>>> You cover a related issue in your "Homogenous Developers" section. But I
>>>>> think
>>>>> building on the "Community" section of the current state to call this
>>>>> out
>>>>> as an
>>>>> independent issue is worthwhile.
>>>>>
>>>>>
>>>>>   possible.  This environment includes widely accessible source code
>>>>>>> repositories, published artifacts, ticket tracking, and extensive
>>>>>>> documentation. We also encourage contributions and frequent debate and
>>>>>>> hold regular, collaborative discussions through e-mail, chat rooms,
>>>>>>> and in-person meet-ups.
>>>>>>>
>>>>>> Do you anticipate any difficulties moving these established
>>>>> communication
>>>>> mechanisms to ASF public lists?
>>>>>
>>>>>   === Documentation ===
>>>>>>> At this time there is no NiFi documentation on the web.  However, we
>>>>>>> have extensive documentation included within the application that
>>>>>>> details usage of the many functions.  We will be rapidly expanding the
>>>>>>> available documentation to cover things like installation, developer
>>>>>>> guide, frequently asked questions, best practices, and more.  This
>>>>>>> documentation will be posted to the NiFi wiki at apache.org.
>>>>>>>
>>>>>> I love projects that start with documentation. :)
>>>>> I don't think the proposal needs to include that the documentation will
>>>>>
>>>> be
>>>>
>>>>> posted
>>>>> to the NiFi wiki, since that's an implementation detail. Just say "this
>>>>> documentation
>>>>> will be made available via the NiFi project's use of incubator infra."
>>>>>
>>>>> (I'll save detail for the eventual dev@ list, but you should strongly
>>>>> consider not
>>>>> using the wiki to host this documentation.)
>>>>>
>>>>> -Sean
>>>>>
>>>>> On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland<br...@cloudera.com>
>>>>>
>>>> wrote:
>>>>
>>>>> Hi Joe,
>>>>>> I know you've done a tremendous amount of work to make this happen so I
>>>>>>
>>>>> am
>>>>>
>>>>>> extremely happy this is *finally* making it's way to the incubator!
>>>>>>
>>>>>> I look forward to helping in anyway I can.
>>>>>>
>>>>>> Cheers!
>>>>>> Brock
>>>>>>
>>>>>> On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980)<
>>>>>> chris.a.mattmann@jpl.nasa.gov>   wrote:
>>>>>>
>>>>>>   This is *fan freakin¹ tastic* Sounds like an awesome  project and
>>>>>>> glad to hear a relationship to Tika! Awesome to see more government
>>>>>>> projects coming into the ASF!
>>>>>>>
>>>>>>> you already have a great set of mentors and I don¹t really have more
>>>>>>> time on my plate, but really happy and will try and monitor and help
>>>>>>> on the lists.
>>>>>>>
>>>>>>> Cheers!
>>>>>>>
>>>>>>> Chris
>>>>>>>
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> Chris Mattmann, Ph.D.
>>>>>>> Chief Architect
>>>>>>> Instrument Software and Science Data Systems Section (398)
>>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>> Office: 168-519, Mailstop: 168-527
>>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>>> WWW:  http://sunset.usc.edu/~mattmann/
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> Adjunct Associate Professor, Computer Science Department
>>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Joe Witt<jo...@gmail.com>
>>>>>>> Reply-To: "general@incubator.apache.org"<
>>>>>>>
>>>>>> general@incubator.apache.org
>>>>> Date: Thursday, November 20, 2014 at 3:02 AM
>>>>>>> To: "general@incubator.apache.org"<ge...@incubator.apache.org>
>>>>>>> Subject: [PROPOSAL] NiFi for Incubation
>>>>>>>
>>>>>>>   Hello,
>>>>>>>>
>>>>>>>> I would like to propose NiFi as an Apache Incubator Project.
>>>>>>>>
>>>>>>>> In addition to the copy provided below the Wiki version of the
>>>>>>>> proposal can be found here:
>>>>>>>> http://wiki.apache.org/incubator/NiFiProposal
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> Joe
>>>>>>>>
>>>>>>>>
>>>>>>>> = NiFi Proposal =
>>>>>>>>
>>>>>>>> == Abstract ==
>>>>>>>> NiFi is a dataflow system based on the concepts of flow-based
>>>>>>>>
>>>>>>> programming.
>>>>>>> == Proposal ==
>>>>>>>> NiFi supports powerful and scalable directed graphs of data routing,
>>>>>>>> transformation, and system mediation logic.  Some of the high-level
>>>>>>>> capabilities and objectives of NiFi include:
>>>>>>>>    * Web-based user interface for seamless experience between design,
>>>>>>>> control, feedback, and monitoring of data flows
>>>>>>>>    * Highly configurable along several dimensions of quality of
>>>>>>>>
>>>>>>> service
>>>>> such as loss tolerant versus guaranteed delivery, low latency versus
>>>>>>>> high throughput, and priority based queuing
>>>>>>>>    * Fine-grained data provenance for all data received, forked,
>>>>>>>> joined, cloned, modified, sent, and ultimately dropped as data
>>>>>>>>
>>>>>>> reaches
>>>>> its configured end-state
>>>>>>>>    * Component-based extension model along well defined interfaces
>>>>>>>> enabling rapid development and effective testing
>>>>>>>>
>>>>>>>> == Background ==
>>>>>>>> Reliable and effective dataflow between systems can be difficult
>>>>>>>> whether you're running scripts on a laptop or have a massive
>>>>>>>> distributed computing system operated by numerous teams and
>>>>>>>> organizations.  As the volume and rate of data grows and as the
>>>>>>>>
>>>>>>> number
>>>>> of systems, protocols, and formats increase and evolve so too does
>>>>>>> the
>>>>> complexity and need for greater insight and agility.  These are the
>>>>>>>> dataflow challenges that NiFi was built to tackle.
>>>>>>>>
>>>>>>>> NiFi is designed in a manner consistent with the core concepts
>>>>>>>> described in flow-based programming as originally documented by J.
>>>>>>>> Paul Morrison in the 1970s.  This model lends itself well to visual
>>>>>>>> diagramming, concurrency, componentization, testing, and reuse.  In
>>>>>>>> addition to staying close to the fundamentals of flow-based
>>>>>>>> programming, NiFi provides integration system specific features such
>>>>>>>> as: guaranteed delivery; back pressure; ability to gracefully handle
>>>>>>>> backlogs and data surges; and an operator interface that enables
>>>>>>>> on-the-fly data flow generation, modification, and observation.
>>>>>>>>
>>>>>>>> == Rationale ==
>>>>>>>> NiFi provides a reliable, scalable, manageable and accountable
>>>>>>>> platform for developers and technical staff to create and evolve
>>>>>>>> powerful data flows.  Such a system is useful in many contexts
>>>>>>>> including large-scale enterprise integration, interaction with cloud
>>>>>>>> services and frameworks, business to business, intra-departmental,
>>>>>>>>
>>>>>>> and
>>>>> inter-departmental flows.  NiFi fits well within the Apache Software
>>>>>>>> Foundation (ASF) family as it depends on numerous ASF projects and
>>>>>>>> integrates with several others.  We also anticipate developing
>>>>>>>> extensions for several other ASF projects such as Cassandra, Kafka,
>>>>>>>> and Storm in the near future.
>>>>>>>>
>>>>>>>> == Initial Goals ==
>>>>>>>>    * Ensure all dependencies are compliant with Apache License
>>>>>>>>
>>>>>>> version
>>>>> 2.0 and all that all code and documentation artifacts have the
>>>>>>> correct
>>>>> Apache licensing markings and notice.
>>>>>>>>    * Establish a formal release process and schedule, allowing for
>>>>>>>> dependable release cycles in a manner consistent with the Apache
>>>>>>>> development process.
>>>>>>>>    * Determine and establish a mechanism, possibly including a
>>>>>>>> sub-project construct, that allows for extensions to the core
>>>>>>>> application to occur at a pace that differs from the core
>>>>>>>>
>>>>>>> application
>>>>> itself.
>>>>>>>> == Current Status ==
>>>>>>>> === Meritocracy ===
>>>>>>>> An integration platform is only as good as its ability to integrate
>>>>>>>> systems in a reliable, timely, and repeatable manner.  The same can
>>>>>>>>
>>>>>>> be
>>>>> said of its ability to attract talent and a variety of perspectives
>>>>>>> as
>>>>> integration systems by their nature are always evolving.  We will
>>>>>>>> actively seek help and encourage promotion of influence in the
>>>>>>>>
>>>>>>> project
>>>>> through meritocracy.
>>>>>>>> === Community ===
>>>>>>>> Over the past several years, NiFi has developed a strong community
>>>>>>>>
>>>>>>> of
>>>>> both developers and operators within the U.S. government.  We look
>>>>>>>> forward to helping grow this to a broader base of industries.
>>>>>>>>
>>>>>>>> === Core Developers ===
>>>>>>>> The initial core developers are employed by the National Security
>>>>>>>> Agency and defense contractors.  We will work to grow the community
>>>>>>>> among a more diverse set of developers and industries.
>>>>>>>>
>>>>>>>> === Alignment ===
>>>>>>>>
>>>>>>>>  From its inception, NiFi was developed with an open source
>>>>>> philosophy
>>>>> in mind and with the hopes of eventually being truly open sourced.
>>>>>>>> The Apache way is consistent with the approach we have taken to
>>>>>>>>
>>>>>>> date.
>>>>> The ASF clearly provides a mature and effective environment for
>>>>>>>> successful development as is evident across the spectrum of
>>>>>>>>
>>>>>>> well-known
>>>>> projects.  Further, NiFi depends on numerous ASF libraries and
>>>>>>>> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
>>>>>>>> HttpClient, Jakarta and Maven.  We also anticipate extensions and
>>>>>>>> dependencies with several more ASF projects, including Accumulo,
>>>>>>>>
>>>>>>> Avro,
>>>>> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
>>>>>>>> This existing alignment with Apache and the desired community makes
>>>>>>>> the Apache Incubator a good fit for NiFi.
>>>>>>>>
>>>>>>>> == Known Risks ==
>>>>>>>> === Orphaned Products ===
>>>>>>>> Risk of orphaning is minimal.  The project user and developer base
>>>>>>>>
>>>>>>> is
>>>>> substantial, growing, and there is already extensive operational use
>>>>>>>> of NiFi.
>>>>>>>>
>>>>>>>> === Inexperience with Open Source ===
>>>>>>>> The initial committers to NiFi have limited experience with true
>>>>>>>>
>>>>>>> open
>>>>> source software development.  However, despite the project origins
>>>>>>>> being from closed source development we have modelled our behavior
>>>>>>>>
>>>>>>> and
>>>>> community development on The Apache Way to the greatest extent
>>>>>>>> possible.  This environment includes widely accessible source code
>>>>>>>> repositories, published artifacts, ticket tracking, and extensive
>>>>>>>> documentation. We also encourage contributions and frequent debate
>>>>>>>>
>>>>>>> and
>>>>> hold regular, collaborative discussions through e-mail, chat rooms,
>>>>>>>> and in-person meet-ups.  We are committed to the ideals of open
>>>>>>>>
>>>>>>> source
>>>>> software and will eagerly seek out mentors and sponsors who can help
>>>>>>>> us quickly come up to speed.
>>>>>>>>
>>>>>>>> === Homogenous Developers ===
>>>>>>>> The initial committers of NiFi come from a limited set of entities
>>>>>>>> though we are committed to recruiting and developing additional
>>>>>>>> committers from a broad spectrum of industries and backgrounds.
>>>>>>>>
>>>>>>>> === Reliance on Salaried Developers ===
>>>>>>>> We expect NiFi development to continue on salaried time and through
>>>>>>>> volunteer time.  The initial committers are paid by their employers
>>>>>>>>
>>>>>>> to
>>>>> contribute to this project.  We are committed to developing and
>>>>>>>> recruiting participation from developers both salaried and
>>>>>>>> non-salaried.
>>>>>>>>
>>>>>>>> === Relationship with other Apache Projects ===
>>>>>>>> As described in the alignment section, NiFi is already heavily
>>>>>>>> dependent on other ASF projects and we anticipate further dependence
>>>>>>>> and integration with new and emerging projects in the Apache family.
>>>>>>>>
>>>>>>>> === An Excessive Fascination with the Apache Brand ===
>>>>>>>> We respect the laudable Apache brand and that is certainly a factor
>>>>>>>>
>>>>>>> in
>>>>> the decision to propose NiFi for the Apache Incubator.  We intend to
>>>>>>>> provide a great deal of energy and capability to the ASF through
>>>>>>>>
>>>>>>> this
>>>>> project.  We will be sensitive to and respectful of any overuse of
>>>>>>> the
>>>>> Apache brand and ensure our focus remains on how we benefit the
>>>>>>> Apache
>>>>> community.
>>>>>>>> === Documentation ===
>>>>>>>> At this time there is no NiFi documentation on the web.  However, we
>>>>>>>> have extensive documentation included within the application that
>>>>>>>> details usage of the many functions.  We will be rapidly expanding
>>>>>>>>
>>>>>>> the
>>>>> available documentation to cover things like installation, developer
>>>>>>>> guide, frequently asked questions, best practices, and more.  This
>>>>>>>> documentation will be posted to the NiFi wiki at apache.org.
>>>>>>>>
>>>>>>>> == Initial Source ==
>>>>>>>> NiFi has been in active development since late 2006 with
>>>>>>>>
>>>>>>> contributions
>>>>> >from dozens of developers and feedback from hundreds of users and
>>>>>>>> developers.  The core codebase is written in Java and includes
>>>>>>>> detailed Javadocs and feature documentation.
>>>>>>>>
>>>>>>>> == Source and Intellectual Property Submission ==
>>>>>>>> Previously referred to as Niagarafiles, the NiFi code and
>>>>>>>> documentation materials will be submitted by the National Security
>>>>>>>> Agency.  NiFi has been developed by a mix of government employees
>>>>>>>>
>>>>>>> and
>>>>> private companies under government contract.  Material developed by
>>>>>>>> the government employees is in the public domain and no U.S.
>>>>>>>>
>>>>>>> copyright
>>>>> exists in works of the federal government.  For the contractor
>>>>>>>> developed material in the initial submission, the U.S. Government
>>>>>>>>
>>>>>>> has
>>>>> sufficient authority to open source per DFARS 252.227-7014.  NSA
>>>>>>> has
>>>>> submitted the Software Grant Agreement and Corporate Contributor
>>>>>>>> License Agreement to the Apache Software Foundation.
>>>>>>>>
>>>>>>>> == External Dependencies ==
>>>>>>>> We have at least one dependency on an LGPL library which we will
>>>>>>>> promptly address.  Otherwise, we believe all current dependencies
>>>>>>>>
>>>>>>> are
>>>>> compatible with the ASF guidelines.  Our dependency licenses come
>>>>>>> from
>>>>> the following license styles:  Apache v 2.0, BSD, Public Domain,
>>>>>>>> Eclipse Public v1, MIT, CDDL v1.
>>>>>>>>
>>>>>>>> == Cryptography ==
>>>>>>>> Consistent with http://www.apache.org/licenses/exports/ we believe
>>>>>>>> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
>>>>>>>> cryptographic algorithms but is designed to use algorithms provided
>>>>>>>>
>>>>>>> by
>>>>> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
>>>>>>>> These cryptographic algorithm providers are used to support SSL,
>>>>>>>> SSH/SFTP, and the encryption and decryption of sensitive properties.
>>>>>>>> In the event that it becomes necessary we will engage with
>>>>>>>>
>>>>>>> appropriate
>>>>> Apache members to ensure we file any necessary paperwork or
>>>>>>> clarified
>>>>> any cryptographic export license concerns.
>>>>>>>> == Required Resources ==
>>>>>>>> === Mailing Lists ===
>>>>>>>>    * user@nifi.incubator.apache.org
>>>>>>>>    * dev@nifi.incubator.apache.org
>>>>>>>>    * private@nifi.incubator.apache.org
>>>>>>>>    * commits@nifi.incubator.apache.org
>>>>>>>>
>>>>>>>> === Source Control ===
>>>>>>>> NiFi requests use of Git for source control
>>>>>>>> (git://git.apache.org/nifi.git).  We request a writeable Git repo
>>>>>>>>
>>>>>>> for
>>>>> NiFi with mirroring to be setup to Github through INFRA.  We request
>>>>>>>> sponsor Benson Margulies (bimargulies) to assist with creating the
>>>>>>>> INFRA ticket for this.
>>>>>>>>
>>>>>>>> === Issue Tracking ===
>>>>>>>> JIRA NiFi (NIFI)
>>>>>>>>
>>>>>>>> === Initial Committers ===
>>>>>>>>    * Brandon Devries<brandon.devries at gmail dot com>
>>>>>>>>    * Matt Gilman<matt.c.gilman at gmail dot com>
>>>>>>>>    * Tony Kurc<trkurc at gmail dot com>
>>>>>>>>    * Mark Payne<markap14 at hotmail dot com>
>>>>>>>>    * Adam Taft<adam at adamtaft dot com>
>>>>>>>>    * Joseph Witt<joewitt at gmail dot com>
>>>>>>>>
>>>>>>>> === Affiliations ===
>>>>>>>>    * Brandon Devries (Requitest, Inc.)
>>>>>>>>    * Matt Gilman (Raytheon)
>>>>>>>>    * Tony Kurc (National Security Agency)
>>>>>>>>    * Mark Payne (Sotera Defense Solutions, Inc.)
>>>>>>>>    * Adam Taft (Requitest, Inc.)
>>>>>>>>    * Joseph Witt (National Security Agency)
>>>>>>>>
>>>>>>>> == Sponsors ==
>>>>>>>> === Champion ===
>>>>>>>>    * Benson Margulies (Basis Technology)<bimargulies at apache dot
>>>>>>>>
>>>>>>> org>
>>>>>> === Nominated Mentors ===
>>>>>>>>    * Drew Farris (Booz Allen Hamilton)<drew at apache dot org>
>>>>>>>>    * Brock Noland (Cloudera)<brock at apache dot org>
>>>>>>>>    * Billie Rinaldi (Hortonworks)<billie at apache dot org>
>>>>>>>>
>>>>>>>> === Sponsoring Entity ===
>>>>>>>> We request the Apache Incubator to sponsor this project.
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> --
>>>>> Sean
>>>>>
>>>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [PROPOSAL] NiFi for Incubation

[Andrew Purtell <ap...@apache.org>]

I didn't make an assertion *about* Accumulo, what I said exactly was:

        "that Accumulo may or may not suffer from "bifurcation" is
debatable"

I assert there are two sides to that argument (I've heard them both)​​,
merely to suggest that Sean's concerns should not be dismissed as they
were.


On Fri, Nov 21, 2014 at 2:11 PM, Josh Elser <el...@apache.org> wrote:

> I would tend to disagree with your assertion about Accumulo, but this is
> neither the time nor the place to discuss Accumulo's strengths and
> weaknesses.
>
> I'll make the assumption that request to enter incubation is a strong
> intent to grow outside of the project's current home.
>
> +1 (non-binding) for entry
>
>
> Andrew Purtell wrote:
>
>>  From what I understand, that Accumulo may or may not suffer from
>> "bifurcation" is debatable despite this blanket claim otherwise.
>>
>> Regardless, Sean's point is valid, even if only a theoretical concern for
>> NiFi at this point. If the podling is interested in growing outside
>> community I'd be happy to sign on as a mentor to help that happen.
>>
>> On Thu, Nov 20, 2014 at 3:32 AM, Benson Margulies<bi...@gmail.com>
>> wrote:
>>
>>  Sean,
>>>
>>> The precedent of Accumulo is that the govt people and agencies involved
>>> are
>>> ready and able to have their staff collaborate openly in an Apache
>>> community. There's no need to contemplate bifurcation; we have this
>>> proposal because the management recognizes that this collaboration
>>> produces
>>> better stuff that solves more problems than the 'inside the tent'
>>> alternative.
>>>
>>> --benson
>>>
>>>
>>> On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey<bu...@cloudera.com>
>>> wrote:
>>>
>>>  I'm really excited to see NiFi come to the incubator; it'd be a great
>>>> addition to the ASF.
>>>>
>>>> A few points in the proposal:
>>>>
>>>>  == Initial Goals ==
>>>>>>
>>>>> One of these should be to grow the community outside of the current
>>>>
>>> niche,
>>>
>>>> IMHO.
>>>>
>>>> More on this below under "orphaned projects"
>>>>
>>>>    * Determine and establish a mechanism, possibly including a
>>>>>> sub-project construct, that allows for extensions to the core
>>>>>> application to occur at a pace that differs from the core application
>>>>>> itself.
>>>>>>
>>>>> I don't think the proposal needs to include the "e.g. with
>>>> sub-projects"
>>>> part. Just noting
>>>> that your goals in the incubator are to address the need to have
>>>>
>>> different
>>>
>>>> release cycles
>>>> for core and extensions is sufficient.
>>>>
>>>>
>>>>  === Community ===
>>>>>> Over the past several years, NiFi has developed a strong community of
>>>>>> both developers and operators within the U.S. government.  We look
>>>>>> forward to helping grow this to a broader base of industries.
>>>>>>
>>>>>>  How much, if any, of this community do you expect to engage via the
>>>> customary project
>>>> lists once NiFi is established within the ASF? Will the project be able
>>>>
>>> to
>>>
>>>> leverage this
>>>> established group?
>>>>
>>>>
>>>>  === Orphaned Products ===
>>>>>> Risk of orphaning is minimal.  The project user and developer base is
>>>>>> substantial, growing, and there is already extensive operational use
>>>>>> of NiFi.
>>>>>>
>>>>> Given that the established base is internal to the U.S. government, I'd
>>>> encourage the
>>>> podling to consider the risk of a bifurcated project should a
>>>> substantial
>>>> outside
>>>> community fail to emerge or if those internal users should fail to
>>>> engage
>>>> with the
>>>> outside community.
>>>>
>>>> You cover a related issue in your "Homogenous Developers" section. But I
>>>> think
>>>> building on the "Community" section of the current state to call this
>>>> out
>>>> as an
>>>> independent issue is worthwhile.
>>>>
>>>>
>>>>  possible.  This environment includes widely accessible source code
>>>>>> repositories, published artifacts, ticket tracking, and extensive
>>>>>> documentation. We also encourage contributions and frequent debate and
>>>>>> hold regular, collaborative discussions through e-mail, chat rooms,
>>>>>> and in-person meet-ups.
>>>>>>
>>>>> Do you anticipate any difficulties moving these established
>>>> communication
>>>> mechanisms to ASF public lists?
>>>>
>>>>  === Documentation ===
>>>>>> At this time there is no NiFi documentation on the web.  However, we
>>>>>> have extensive documentation included within the application that
>>>>>> details usage of the many functions.  We will be rapidly expanding the
>>>>>> available documentation to cover things like installation, developer
>>>>>> guide, frequently asked questions, best practices, and more.  This
>>>>>> documentation will be posted to the NiFi wiki at apache.org.
>>>>>>
>>>>> I love projects that start with documentation. :)
>>>>
>>>> I don't think the proposal needs to include that the documentation will
>>>>
>>> be
>>>
>>>> posted
>>>> to the NiFi wiki, since that's an implementation detail. Just say "this
>>>> documentation
>>>> will be made available via the NiFi project's use of incubator infra."
>>>>
>>>> (I'll save detail for the eventual dev@ list, but you should strongly
>>>> consider not
>>>> using the wiki to host this documentation.)
>>>>
>>>> -Sean
>>>>
>>>> On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland<br...@cloudera.com>
>>>>
>>> wrote:
>>>
>>>> Hi Joe,
>>>>>
>>>>> I know you've done a tremendous amount of work to make this happen so I
>>>>>
>>>> am
>>>>
>>>>> extremely happy this is *finally* making it's way to the incubator!
>>>>>
>>>>> I look forward to helping in anyway I can.
>>>>>
>>>>> Cheers!
>>>>> Brock
>>>>>
>>>>> On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980)<
>>>>> chris.a.mattmann@jpl.nasa.gov>  wrote:
>>>>>
>>>>>  This is *fan freakin¹ tastic* Sounds like an awesome  project and
>>>>>> glad to hear a relationship to Tika! Awesome to see more government
>>>>>> projects coming into the ASF!
>>>>>>
>>>>>> you already have a great set of mentors and I don¹t really have more
>>>>>> time on my plate, but really happy and will try and monitor and help
>>>>>> on the lists.
>>>>>>
>>>>>> Cheers!
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Chris Mattmann, Ph.D.
>>>>>> Chief Architect
>>>>>> Instrument Software and Science Data Systems Section (398)
>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>> Office: 168-519, Mailstop: 168-527
>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>> WWW:  http://sunset.usc.edu/~mattmann/
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Adjunct Associate Professor, Computer Science Department
>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Joe Witt<jo...@gmail.com>
>>>>>> Reply-To: "general@incubator.apache.org"<
>>>>>>
>>>>> general@incubator.apache.org
>>>
>>>> Date: Thursday, November 20, 2014 at 3:02 AM
>>>>>> To: "general@incubator.apache.org"<ge...@incubator.apache.org>
>>>>>> Subject: [PROPOSAL] NiFi for Incubation
>>>>>>
>>>>>>  Hello,
>>>>>>>
>>>>>>>
>>>>>>> I would like to propose NiFi as an Apache Incubator Project.
>>>>>>>
>>>>>>> In addition to the copy provided below the Wiki version of the
>>>>>>> proposal can be found here:
>>>>>>> http://wiki.apache.org/incubator/NiFiProposal
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Joe
>>>>>>>
>>>>>>>
>>>>>>> = NiFi Proposal =
>>>>>>>
>>>>>>> == Abstract ==
>>>>>>> NiFi is a dataflow system based on the concepts of flow-based
>>>>>>>
>>>>>> programming.
>>>>>
>>>>>> == Proposal ==
>>>>>>> NiFi supports powerful and scalable directed graphs of data routing,
>>>>>>> transformation, and system mediation logic.  Some of the high-level
>>>>>>> capabilities and objectives of NiFi include:
>>>>>>>   * Web-based user interface for seamless experience between design,
>>>>>>> control, feedback, and monitoring of data flows
>>>>>>>   * Highly configurable along several dimensions of quality of
>>>>>>>
>>>>>> service
>>>
>>>> such as loss tolerant versus guaranteed delivery, low latency versus
>>>>>>> high throughput, and priority based queuing
>>>>>>>   * Fine-grained data provenance for all data received, forked,
>>>>>>> joined, cloned, modified, sent, and ultimately dropped as data
>>>>>>>
>>>>>> reaches
>>>
>>>> its configured end-state
>>>>>>>   * Component-based extension model along well defined interfaces
>>>>>>> enabling rapid development and effective testing
>>>>>>>
>>>>>>> == Background ==
>>>>>>> Reliable and effective dataflow between systems can be difficult
>>>>>>> whether you're running scripts on a laptop or have a massive
>>>>>>> distributed computing system operated by numerous teams and
>>>>>>> organizations.  As the volume and rate of data grows and as the
>>>>>>>
>>>>>> number
>>>
>>>> of systems, protocols, and formats increase and evolve so too does
>>>>>>>
>>>>>> the
>>>
>>>> complexity and need for greater insight and agility.  These are the
>>>>>>> dataflow challenges that NiFi was built to tackle.
>>>>>>>
>>>>>>> NiFi is designed in a manner consistent with the core concepts
>>>>>>> described in flow-based programming as originally documented by J.
>>>>>>> Paul Morrison in the 1970s.  This model lends itself well to visual
>>>>>>> diagramming, concurrency, componentization, testing, and reuse.  In
>>>>>>> addition to staying close to the fundamentals of flow-based
>>>>>>> programming, NiFi provides integration system specific features such
>>>>>>> as: guaranteed delivery; back pressure; ability to gracefully handle
>>>>>>> backlogs and data surges; and an operator interface that enables
>>>>>>> on-the-fly data flow generation, modification, and observation.
>>>>>>>
>>>>>>> == Rationale ==
>>>>>>> NiFi provides a reliable, scalable, manageable and accountable
>>>>>>> platform for developers and technical staff to create and evolve
>>>>>>> powerful data flows.  Such a system is useful in many contexts
>>>>>>> including large-scale enterprise integration, interaction with cloud
>>>>>>> services and frameworks, business to business, intra-departmental,
>>>>>>>
>>>>>> and
>>>
>>>> inter-departmental flows.  NiFi fits well within the Apache Software
>>>>>>> Foundation (ASF) family as it depends on numerous ASF projects and
>>>>>>> integrates with several others.  We also anticipate developing
>>>>>>> extensions for several other ASF projects such as Cassandra, Kafka,
>>>>>>> and Storm in the near future.
>>>>>>>
>>>>>>> == Initial Goals ==
>>>>>>>   * Ensure all dependencies are compliant with Apache License
>>>>>>>
>>>>>> version
>>>
>>>> 2.0 and all that all code and documentation artifacts have the
>>>>>>>
>>>>>> correct
>>>
>>>> Apache licensing markings and notice.
>>>>>>>   * Establish a formal release process and schedule, allowing for
>>>>>>> dependable release cycles in a manner consistent with the Apache
>>>>>>> development process.
>>>>>>>   * Determine and establish a mechanism, possibly including a
>>>>>>> sub-project construct, that allows for extensions to the core
>>>>>>> application to occur at a pace that differs from the core
>>>>>>>
>>>>>> application
>>>
>>>> itself.
>>>>>>>
>>>>>>> == Current Status ==
>>>>>>> === Meritocracy ===
>>>>>>> An integration platform is only as good as its ability to integrate
>>>>>>> systems in a reliable, timely, and repeatable manner.  The same can
>>>>>>>
>>>>>> be
>>>
>>>> said of its ability to attract talent and a variety of perspectives
>>>>>>>
>>>>>> as
>>>
>>>> integration systems by their nature are always evolving.  We will
>>>>>>> actively seek help and encourage promotion of influence in the
>>>>>>>
>>>>>> project
>>>
>>>> through meritocracy.
>>>>>>>
>>>>>>> === Community ===
>>>>>>> Over the past several years, NiFi has developed a strong community
>>>>>>>
>>>>>> of
>>>
>>>> both developers and operators within the U.S. government.  We look
>>>>>>> forward to helping grow this to a broader base of industries.
>>>>>>>
>>>>>>> === Core Developers ===
>>>>>>> The initial core developers are employed by the National Security
>>>>>>> Agency and defense contractors.  We will work to grow the community
>>>>>>> among a more diverse set of developers and industries.
>>>>>>>
>>>>>>> === Alignment ===
>>>>>>>
>>>>>> > From its inception, NiFi was developed with an open source
>>>>>>
>>>>> philosophy
>>>
>>>> in mind and with the hopes of eventually being truly open sourced.
>>>>>>> The Apache way is consistent with the approach we have taken to
>>>>>>>
>>>>>> date.
>>>
>>>> The ASF clearly provides a mature and effective environment for
>>>>>>> successful development as is evident across the spectrum of
>>>>>>>
>>>>>> well-known
>>>
>>>> projects.  Further, NiFi depends on numerous ASF libraries and
>>>>>>> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
>>>>>>> HttpClient, Jakarta and Maven.  We also anticipate extensions and
>>>>>>> dependencies with several more ASF projects, including Accumulo,
>>>>>>>
>>>>>> Avro,
>>>
>>>> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
>>>>>>> This existing alignment with Apache and the desired community makes
>>>>>>> the Apache Incubator a good fit for NiFi.
>>>>>>>
>>>>>>> == Known Risks ==
>>>>>>> === Orphaned Products ===
>>>>>>> Risk of orphaning is minimal.  The project user and developer base
>>>>>>>
>>>>>> is
>>>
>>>> substantial, growing, and there is already extensive operational use
>>>>>>> of NiFi.
>>>>>>>
>>>>>>> === Inexperience with Open Source ===
>>>>>>> The initial committers to NiFi have limited experience with true
>>>>>>>
>>>>>> open
>>>
>>>> source software development.  However, despite the project origins
>>>>>>> being from closed source development we have modelled our behavior
>>>>>>>
>>>>>> and
>>>
>>>> community development on The Apache Way to the greatest extent
>>>>>>> possible.  This environment includes widely accessible source code
>>>>>>> repositories, published artifacts, ticket tracking, and extensive
>>>>>>> documentation. We also encourage contributions and frequent debate
>>>>>>>
>>>>>> and
>>>
>>>> hold regular, collaborative discussions through e-mail, chat rooms,
>>>>>>> and in-person meet-ups.  We are committed to the ideals of open
>>>>>>>
>>>>>> source
>>>
>>>> software and will eagerly seek out mentors and sponsors who can help
>>>>>>> us quickly come up to speed.
>>>>>>>
>>>>>>> === Homogenous Developers ===
>>>>>>> The initial committers of NiFi come from a limited set of entities
>>>>>>> though we are committed to recruiting and developing additional
>>>>>>> committers from a broad spectrum of industries and backgrounds.
>>>>>>>
>>>>>>> === Reliance on Salaried Developers ===
>>>>>>> We expect NiFi development to continue on salaried time and through
>>>>>>> volunteer time.  The initial committers are paid by their employers
>>>>>>>
>>>>>> to
>>>
>>>> contribute to this project.  We are committed to developing and
>>>>>>> recruiting participation from developers both salaried and
>>>>>>> non-salaried.
>>>>>>>
>>>>>>> === Relationship with other Apache Projects ===
>>>>>>> As described in the alignment section, NiFi is already heavily
>>>>>>> dependent on other ASF projects and we anticipate further dependence
>>>>>>> and integration with new and emerging projects in the Apache family.
>>>>>>>
>>>>>>> === An Excessive Fascination with the Apache Brand ===
>>>>>>> We respect the laudable Apache brand and that is certainly a factor
>>>>>>>
>>>>>> in
>>>
>>>> the decision to propose NiFi for the Apache Incubator.  We intend to
>>>>>>> provide a great deal of energy and capability to the ASF through
>>>>>>>
>>>>>> this
>>>
>>>> project.  We will be sensitive to and respectful of any overuse of
>>>>>>>
>>>>>> the
>>>
>>>> Apache brand and ensure our focus remains on how we benefit the
>>>>>>>
>>>>>> Apache
>>>
>>>> community.
>>>>>>>
>>>>>>> === Documentation ===
>>>>>>> At this time there is no NiFi documentation on the web.  However, we
>>>>>>> have extensive documentation included within the application that
>>>>>>> details usage of the many functions.  We will be rapidly expanding
>>>>>>>
>>>>>> the
>>>
>>>> available documentation to cover things like installation, developer
>>>>>>> guide, frequently asked questions, best practices, and more.  This
>>>>>>> documentation will be posted to the NiFi wiki at apache.org.
>>>>>>>
>>>>>>> == Initial Source ==
>>>>>>> NiFi has been in active development since late 2006 with
>>>>>>>
>>>>>> contributions
>>>
>>>> >from dozens of developers and feedback from hundreds of users and
>>>>>>
>>>>>>> developers.  The core codebase is written in Java and includes
>>>>>>> detailed Javadocs and feature documentation.
>>>>>>>
>>>>>>> == Source and Intellectual Property Submission ==
>>>>>>> Previously referred to as Niagarafiles, the NiFi code and
>>>>>>> documentation materials will be submitted by the National Security
>>>>>>> Agency.  NiFi has been developed by a mix of government employees
>>>>>>>
>>>>>> and
>>>
>>>> private companies under government contract.  Material developed by
>>>>>>> the government employees is in the public domain and no U.S.
>>>>>>>
>>>>>> copyright
>>>
>>>> exists in works of the federal government.  For the contractor
>>>>>>> developed material in the initial submission, the U.S. Government
>>>>>>>
>>>>>> has
>>>
>>>> sufficient authority to open source per DFARS 252.227-7014.  NSA
>>>>>>>
>>>>>> has
>>>
>>>> submitted the Software Grant Agreement and Corporate Contributor
>>>>>>> License Agreement to the Apache Software Foundation.
>>>>>>>
>>>>>>> == External Dependencies ==
>>>>>>> We have at least one dependency on an LGPL library which we will
>>>>>>> promptly address.  Otherwise, we believe all current dependencies
>>>>>>>
>>>>>> are
>>>
>>>> compatible with the ASF guidelines.  Our dependency licenses come
>>>>>>>
>>>>>> from
>>>
>>>> the following license styles:  Apache v 2.0, BSD, Public Domain,
>>>>>>> Eclipse Public v1, MIT, CDDL v1.
>>>>>>>
>>>>>>> == Cryptography ==
>>>>>>> Consistent with http://www.apache.org/licenses/exports/ we believe
>>>>>>> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
>>>>>>> cryptographic algorithms but is designed to use algorithms provided
>>>>>>>
>>>>>> by
>>>
>>>> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
>>>>>>> These cryptographic algorithm providers are used to support SSL,
>>>>>>> SSH/SFTP, and the encryption and decryption of sensitive properties.
>>>>>>> In the event that it becomes necessary we will engage with
>>>>>>>
>>>>>> appropriate
>>>
>>>> Apache members to ensure we file any necessary paperwork or
>>>>>>>
>>>>>> clarified
>>>
>>>> any cryptographic export license concerns.
>>>>>>>
>>>>>>> == Required Resources ==
>>>>>>> === Mailing Lists ===
>>>>>>>   * user@nifi.incubator.apache.org
>>>>>>>   * dev@nifi.incubator.apache.org
>>>>>>>   * private@nifi.incubator.apache.org
>>>>>>>   * commits@nifi.incubator.apache.org
>>>>>>>
>>>>>>> === Source Control ===
>>>>>>> NiFi requests use of Git for source control
>>>>>>> (git://git.apache.org/nifi.git).  We request a writeable Git repo
>>>>>>>
>>>>>> for
>>>
>>>> NiFi with mirroring to be setup to Github through INFRA.  We request
>>>>>>> sponsor Benson Margulies (bimargulies) to assist with creating the
>>>>>>> INFRA ticket for this.
>>>>>>>
>>>>>>> === Issue Tracking ===
>>>>>>> JIRA NiFi (NIFI)
>>>>>>>
>>>>>>> === Initial Committers ===
>>>>>>>   * Brandon Devries<brandon.devries at gmail dot com>
>>>>>>>   * Matt Gilman<matt.c.gilman at gmail dot com>
>>>>>>>   * Tony Kurc<trkurc at gmail dot com>
>>>>>>>   * Mark Payne<markap14 at hotmail dot com>
>>>>>>>   * Adam Taft<adam at adamtaft dot com>
>>>>>>>   * Joseph Witt<joewitt at gmail dot com>
>>>>>>>
>>>>>>> === Affiliations ===
>>>>>>>   * Brandon Devries (Requitest, Inc.)
>>>>>>>   * Matt Gilman (Raytheon)
>>>>>>>   * Tony Kurc (National Security Agency)
>>>>>>>   * Mark Payne (Sotera Defense Solutions, Inc.)
>>>>>>>   * Adam Taft (Requitest, Inc.)
>>>>>>>   * Joseph Witt (National Security Agency)
>>>>>>>
>>>>>>> == Sponsors ==
>>>>>>> === Champion ===
>>>>>>>   * Benson Margulies (Basis Technology)<bimargulies at apache dot
>>>>>>>
>>>>>> org>
>>>>
>>>>> === Nominated Mentors ===
>>>>>>>   * Drew Farris (Booz Allen Hamilton)<drew at apache dot org>
>>>>>>>   * Brock Noland (Cloudera)<brock at apache dot org>
>>>>>>>   * Billie Rinaldi (Hortonworks)<billie at apache dot org>
>>>>>>>
>>>>>>> === Sponsoring Entity ===
>>>>>>> We request the Apache Incubator to sponsor this project.
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Sean
>>>>
>>>>
>>
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>


-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [PROPOSAL] NiFi for Incubation

[Josh Elser <el...@apache.org>]

I would tend to disagree with your assertion about Accumulo, but this is 
neither the time nor the place to discuss Accumulo's strengths and 
weaknesses.

I'll make the assumption that request to enter incubation is a strong 
intent to grow outside of the project's current home.

+1 (non-binding) for entry

Andrew Purtell wrote:
>  From what I understand, that Accumulo may or may not suffer from
> "bifurcation" is debatable despite this blanket claim otherwise.
>
> Regardless, Sean's point is valid, even if only a theoretical concern for
> NiFi at this point. If the podling is interested in growing outside
> community I'd be happy to sign on as a mentor to help that happen.
>
> On Thu, Nov 20, 2014 at 3:32 AM, Benson Margulies<bi...@gmail.com>
> wrote:
>
>> Sean,
>>
>> The precedent of Accumulo is that the govt people and agencies involved are
>> ready and able to have their staff collaborate openly in an Apache
>> community. There's no need to contemplate bifurcation; we have this
>> proposal because the management recognizes that this collaboration produces
>> better stuff that solves more problems than the 'inside the tent'
>> alternative.
>>
>> --benson
>>
>>
>> On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey<bu...@cloudera.com>  wrote:
>>
>>> I'm really excited to see NiFi come to the incubator; it'd be a great
>>> addition to the ASF.
>>>
>>> A few points in the proposal:
>>>
>>>>> == Initial Goals ==
>>> One of these should be to grow the community outside of the current
>> niche,
>>> IMHO.
>>>
>>> More on this below under "orphaned projects"
>>>
>>>>>   * Determine and establish a mechanism, possibly including a
>>>>> sub-project construct, that allows for extensions to the core
>>>>> application to occur at a pace that differs from the core application
>>>>> itself.
>>> I don't think the proposal needs to include the "e.g. with sub-projects"
>>> part. Just noting
>>> that your goals in the incubator are to address the need to have
>> different
>>> release cycles
>>> for core and extensions is sufficient.
>>>
>>>
>>>>> === Community ===
>>>>> Over the past several years, NiFi has developed a strong community of
>>>>> both developers and operators within the U.S. government.  We look
>>>>> forward to helping grow this to a broader base of industries.
>>>>>
>>> How much, if any, of this community do you expect to engage via the
>>> customary project
>>> lists once NiFi is established within the ASF? Will the project be able
>> to
>>> leverage this
>>> established group?
>>>
>>>
>>>>> === Orphaned Products ===
>>>>> Risk of orphaning is minimal.  The project user and developer base is
>>>>> substantial, growing, and there is already extensive operational use
>>>>> of NiFi.
>>> Given that the established base is internal to the U.S. government, I'd
>>> encourage the
>>> podling to consider the risk of a bifurcated project should a substantial
>>> outside
>>> community fail to emerge or if those internal users should fail to engage
>>> with the
>>> outside community.
>>>
>>> You cover a related issue in your "Homogenous Developers" section. But I
>>> think
>>> building on the "Community" section of the current state to call this out
>>> as an
>>> independent issue is worthwhile.
>>>
>>>
>>>>> possible.  This environment includes widely accessible source code
>>>>> repositories, published artifacts, ticket tracking, and extensive
>>>>> documentation. We also encourage contributions and frequent debate and
>>>>> hold regular, collaborative discussions through e-mail, chat rooms,
>>>>> and in-person meet-ups.
>>> Do you anticipate any difficulties moving these established communication
>>> mechanisms to ASF public lists?
>>>
>>>>> === Documentation ===
>>>>> At this time there is no NiFi documentation on the web.  However, we
>>>>> have extensive documentation included within the application that
>>>>> details usage of the many functions.  We will be rapidly expanding the
>>>>> available documentation to cover things like installation, developer
>>>>> guide, frequently asked questions, best practices, and more.  This
>>>>> documentation will be posted to the NiFi wiki at apache.org.
>>> I love projects that start with documentation. :)
>>>
>>> I don't think the proposal needs to include that the documentation will
>> be
>>> posted
>>> to the NiFi wiki, since that's an implementation detail. Just say "this
>>> documentation
>>> will be made available via the NiFi project's use of incubator infra."
>>>
>>> (I'll save detail for the eventual dev@ list, but you should strongly
>>> consider not
>>> using the wiki to host this documentation.)
>>>
>>> -Sean
>>>
>>> On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland<br...@cloudera.com>
>> wrote:
>>>> Hi Joe,
>>>>
>>>> I know you've done a tremendous amount of work to make this happen so I
>>> am
>>>> extremely happy this is *finally* making it's way to the incubator!
>>>>
>>>> I look forward to helping in anyway I can.
>>>>
>>>> Cheers!
>>>> Brock
>>>>
>>>> On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980)<
>>>> chris.a.mattmann@jpl.nasa.gov>  wrote:
>>>>
>>>>> This is *fan freakin¹ tastic* Sounds like an awesome  project and
>>>>> glad to hear a relationship to Tika! Awesome to see more government
>>>>> projects coming into the ASF!
>>>>>
>>>>> you already have a great set of mentors and I don¹t really have more
>>>>> time on my plate, but really happy and will try and monitor and help
>>>>> on the lists.
>>>>>
>>>>> Cheers!
>>>>>
>>>>> Chris
>>>>>
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> Chris Mattmann, Ph.D.
>>>>> Chief Architect
>>>>> Instrument Software and Science Data Systems Section (398)
>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> Office: 168-519, Mailstop: 168-527
>>>>> Email: chris.a.mattmann@nasa.gov
>>>>> WWW:  http://sunset.usc.edu/~mattmann/
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> Adjunct Associate Professor, Computer Science Department
>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Joe Witt<jo...@gmail.com>
>>>>> Reply-To: "general@incubator.apache.org"<
>> general@incubator.apache.org
>>>>> Date: Thursday, November 20, 2014 at 3:02 AM
>>>>> To: "general@incubator.apache.org"<ge...@incubator.apache.org>
>>>>> Subject: [PROPOSAL] NiFi for Incubation
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>>
>>>>>> I would like to propose NiFi as an Apache Incubator Project.
>>>>>>
>>>>>> In addition to the copy provided below the Wiki version of the
>>>>>> proposal can be found here:
>>>>>> http://wiki.apache.org/incubator/NiFiProposal
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Joe
>>>>>>
>>>>>>
>>>>>> = NiFi Proposal =
>>>>>>
>>>>>> == Abstract ==
>>>>>> NiFi is a dataflow system based on the concepts of flow-based
>>>> programming.
>>>>>> == Proposal ==
>>>>>> NiFi supports powerful and scalable directed graphs of data routing,
>>>>>> transformation, and system mediation logic.  Some of the high-level
>>>>>> capabilities and objectives of NiFi include:
>>>>>>   * Web-based user interface for seamless experience between design,
>>>>>> control, feedback, and monitoring of data flows
>>>>>>   * Highly configurable along several dimensions of quality of
>> service
>>>>>> such as loss tolerant versus guaranteed delivery, low latency versus
>>>>>> high throughput, and priority based queuing
>>>>>>   * Fine-grained data provenance for all data received, forked,
>>>>>> joined, cloned, modified, sent, and ultimately dropped as data
>> reaches
>>>>>> its configured end-state
>>>>>>   * Component-based extension model along well defined interfaces
>>>>>> enabling rapid development and effective testing
>>>>>>
>>>>>> == Background ==
>>>>>> Reliable and effective dataflow between systems can be difficult
>>>>>> whether you're running scripts on a laptop or have a massive
>>>>>> distributed computing system operated by numerous teams and
>>>>>> organizations.  As the volume and rate of data grows and as the
>> number
>>>>>> of systems, protocols, and formats increase and evolve so too does
>> the
>>>>>> complexity and need for greater insight and agility.  These are the
>>>>>> dataflow challenges that NiFi was built to tackle.
>>>>>>
>>>>>> NiFi is designed in a manner consistent with the core concepts
>>>>>> described in flow-based programming as originally documented by J.
>>>>>> Paul Morrison in the 1970s.  This model lends itself well to visual
>>>>>> diagramming, concurrency, componentization, testing, and reuse.  In
>>>>>> addition to staying close to the fundamentals of flow-based
>>>>>> programming, NiFi provides integration system specific features such
>>>>>> as: guaranteed delivery; back pressure; ability to gracefully handle
>>>>>> backlogs and data surges; and an operator interface that enables
>>>>>> on-the-fly data flow generation, modification, and observation.
>>>>>>
>>>>>> == Rationale ==
>>>>>> NiFi provides a reliable, scalable, manageable and accountable
>>>>>> platform for developers and technical staff to create and evolve
>>>>>> powerful data flows.  Such a system is useful in many contexts
>>>>>> including large-scale enterprise integration, interaction with cloud
>>>>>> services and frameworks, business to business, intra-departmental,
>> and
>>>>>> inter-departmental flows.  NiFi fits well within the Apache Software
>>>>>> Foundation (ASF) family as it depends on numerous ASF projects and
>>>>>> integrates with several others.  We also anticipate developing
>>>>>> extensions for several other ASF projects such as Cassandra, Kafka,
>>>>>> and Storm in the near future.
>>>>>>
>>>>>> == Initial Goals ==
>>>>>>   * Ensure all dependencies are compliant with Apache License
>> version
>>>>>> 2.0 and all that all code and documentation artifacts have the
>> correct
>>>>>> Apache licensing markings and notice.
>>>>>>   * Establish a formal release process and schedule, allowing for
>>>>>> dependable release cycles in a manner consistent with the Apache
>>>>>> development process.
>>>>>>   * Determine and establish a mechanism, possibly including a
>>>>>> sub-project construct, that allows for extensions to the core
>>>>>> application to occur at a pace that differs from the core
>> application
>>>>>> itself.
>>>>>>
>>>>>> == Current Status ==
>>>>>> === Meritocracy ===
>>>>>> An integration platform is only as good as its ability to integrate
>>>>>> systems in a reliable, timely, and repeatable manner.  The same can
>> be
>>>>>> said of its ability to attract talent and a variety of perspectives
>> as
>>>>>> integration systems by their nature are always evolving.  We will
>>>>>> actively seek help and encourage promotion of influence in the
>> project
>>>>>> through meritocracy.
>>>>>>
>>>>>> === Community ===
>>>>>> Over the past several years, NiFi has developed a strong community
>> of
>>>>>> both developers and operators within the U.S. government.  We look
>>>>>> forward to helping grow this to a broader base of industries.
>>>>>>
>>>>>> === Core Developers ===
>>>>>> The initial core developers are employed by the National Security
>>>>>> Agency and defense contractors.  We will work to grow the community
>>>>>> among a more diverse set of developers and industries.
>>>>>>
>>>>>> === Alignment ===
>>>>> > From its inception, NiFi was developed with an open source
>> philosophy
>>>>>> in mind and with the hopes of eventually being truly open sourced.
>>>>>> The Apache way is consistent with the approach we have taken to
>> date.
>>>>>> The ASF clearly provides a mature and effective environment for
>>>>>> successful development as is evident across the spectrum of
>> well-known
>>>>>> projects.  Further, NiFi depends on numerous ASF libraries and
>>>>>> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
>>>>>> HttpClient, Jakarta and Maven.  We also anticipate extensions and
>>>>>> dependencies with several more ASF projects, including Accumulo,
>> Avro,
>>>>>> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
>>>>>> This existing alignment with Apache and the desired community makes
>>>>>> the Apache Incubator a good fit for NiFi.
>>>>>>
>>>>>> == Known Risks ==
>>>>>> === Orphaned Products ===
>>>>>> Risk of orphaning is minimal.  The project user and developer base
>> is
>>>>>> substantial, growing, and there is already extensive operational use
>>>>>> of NiFi.
>>>>>>
>>>>>> === Inexperience with Open Source ===
>>>>>> The initial committers to NiFi have limited experience with true
>> open
>>>>>> source software development.  However, despite the project origins
>>>>>> being from closed source development we have modelled our behavior
>> and
>>>>>> community development on The Apache Way to the greatest extent
>>>>>> possible.  This environment includes widely accessible source code
>>>>>> repositories, published artifacts, ticket tracking, and extensive
>>>>>> documentation. We also encourage contributions and frequent debate
>> and
>>>>>> hold regular, collaborative discussions through e-mail, chat rooms,
>>>>>> and in-person meet-ups.  We are committed to the ideals of open
>> source
>>>>>> software and will eagerly seek out mentors and sponsors who can help
>>>>>> us quickly come up to speed.
>>>>>>
>>>>>> === Homogenous Developers ===
>>>>>> The initial committers of NiFi come from a limited set of entities
>>>>>> though we are committed to recruiting and developing additional
>>>>>> committers from a broad spectrum of industries and backgrounds.
>>>>>>
>>>>>> === Reliance on Salaried Developers ===
>>>>>> We expect NiFi development to continue on salaried time and through
>>>>>> volunteer time.  The initial committers are paid by their employers
>> to
>>>>>> contribute to this project.  We are committed to developing and
>>>>>> recruiting participation from developers both salaried and
>>>>>> non-salaried.
>>>>>>
>>>>>> === Relationship with other Apache Projects ===
>>>>>> As described in the alignment section, NiFi is already heavily
>>>>>> dependent on other ASF projects and we anticipate further dependence
>>>>>> and integration with new and emerging projects in the Apache family.
>>>>>>
>>>>>> === An Excessive Fascination with the Apache Brand ===
>>>>>> We respect the laudable Apache brand and that is certainly a factor
>> in
>>>>>> the decision to propose NiFi for the Apache Incubator.  We intend to
>>>>>> provide a great deal of energy and capability to the ASF through
>> this
>>>>>> project.  We will be sensitive to and respectful of any overuse of
>> the
>>>>>> Apache brand and ensure our focus remains on how we benefit the
>> Apache
>>>>>> community.
>>>>>>
>>>>>> === Documentation ===
>>>>>> At this time there is no NiFi documentation on the web.  However, we
>>>>>> have extensive documentation included within the application that
>>>>>> details usage of the many functions.  We will be rapidly expanding
>> the
>>>>>> available documentation to cover things like installation, developer
>>>>>> guide, frequently asked questions, best practices, and more.  This
>>>>>> documentation will be posted to the NiFi wiki at apache.org.
>>>>>>
>>>>>> == Initial Source ==
>>>>>> NiFi has been in active development since late 2006 with
>> contributions
>>>>> >from dozens of developers and feedback from hundreds of users and
>>>>>> developers.  The core codebase is written in Java and includes
>>>>>> detailed Javadocs and feature documentation.
>>>>>>
>>>>>> == Source and Intellectual Property Submission ==
>>>>>> Previously referred to as Niagarafiles, the NiFi code and
>>>>>> documentation materials will be submitted by the National Security
>>>>>> Agency.  NiFi has been developed by a mix of government employees
>> and
>>>>>> private companies under government contract.  Material developed by
>>>>>> the government employees is in the public domain and no U.S.
>> copyright
>>>>>> exists in works of the federal government.  For the contractor
>>>>>> developed material in the initial submission, the U.S. Government
>> has
>>>>>> sufficient authority to open source per DFARS 252.227-7014.  NSA
>> has
>>>>>> submitted the Software Grant Agreement and Corporate Contributor
>>>>>> License Agreement to the Apache Software Foundation.
>>>>>>
>>>>>> == External Dependencies ==
>>>>>> We have at least one dependency on an LGPL library which we will
>>>>>> promptly address.  Otherwise, we believe all current dependencies
>> are
>>>>>> compatible with the ASF guidelines.  Our dependency licenses come
>> from
>>>>>> the following license styles:  Apache v 2.0, BSD, Public Domain,
>>>>>> Eclipse Public v1, MIT, CDDL v1.
>>>>>>
>>>>>> == Cryptography ==
>>>>>> Consistent with http://www.apache.org/licenses/exports/ we believe
>>>>>> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
>>>>>> cryptographic algorithms but is designed to use algorithms provided
>> by
>>>>>> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
>>>>>> These cryptographic algorithm providers are used to support SSL,
>>>>>> SSH/SFTP, and the encryption and decryption of sensitive properties.
>>>>>> In the event that it becomes necessary we will engage with
>> appropriate
>>>>>> Apache members to ensure we file any necessary paperwork or
>> clarified
>>>>>> any cryptographic export license concerns.
>>>>>>
>>>>>> == Required Resources ==
>>>>>> === Mailing Lists ===
>>>>>>   * user@nifi.incubator.apache.org
>>>>>>   * dev@nifi.incubator.apache.org
>>>>>>   * private@nifi.incubator.apache.org
>>>>>>   * commits@nifi.incubator.apache.org
>>>>>>
>>>>>> === Source Control ===
>>>>>> NiFi requests use of Git for source control
>>>>>> (git://git.apache.org/nifi.git).  We request a writeable Git repo
>> for
>>>>>> NiFi with mirroring to be setup to Github through INFRA.  We request
>>>>>> sponsor Benson Margulies (bimargulies) to assist with creating the
>>>>>> INFRA ticket for this.
>>>>>>
>>>>>> === Issue Tracking ===
>>>>>> JIRA NiFi (NIFI)
>>>>>>
>>>>>> === Initial Committers ===
>>>>>>   * Brandon Devries<brandon.devries at gmail dot com>
>>>>>>   * Matt Gilman<matt.c.gilman at gmail dot com>
>>>>>>   * Tony Kurc<trkurc at gmail dot com>
>>>>>>   * Mark Payne<markap14 at hotmail dot com>
>>>>>>   * Adam Taft<adam at adamtaft dot com>
>>>>>>   * Joseph Witt<joewitt at gmail dot com>
>>>>>>
>>>>>> === Affiliations ===
>>>>>>   * Brandon Devries (Requitest, Inc.)
>>>>>>   * Matt Gilman (Raytheon)
>>>>>>   * Tony Kurc (National Security Agency)
>>>>>>   * Mark Payne (Sotera Defense Solutions, Inc.)
>>>>>>   * Adam Taft (Requitest, Inc.)
>>>>>>   * Joseph Witt (National Security Agency)
>>>>>>
>>>>>> == Sponsors ==
>>>>>> === Champion ===
>>>>>>   * Benson Margulies (Basis Technology)<bimargulies at apache dot
>>> org>
>>>>>> === Nominated Mentors ===
>>>>>>   * Drew Farris (Booz Allen Hamilton)<drew at apache dot org>
>>>>>>   * Brock Noland (Cloudera)<brock at apache dot org>
>>>>>>   * Billie Rinaldi (Hortonworks)<billie at apache dot org>
>>>>>>
>>>>>> === Sponsoring Entity ===
>>>>>> We request the Apache Incubator to sponsor this project.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>>
>>>>>
>>>
>>>
>>> --
>>> Sean
>>>
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [PROPOSAL] NiFi for Incubation

[Andrew Purtell <ap...@apache.org>]

>From what I understand, that Accumulo may or may not suffer from
"bifurcation" is debatable despite this blanket claim otherwise.

Regardless, Sean's point is valid, even if only a theoretical concern for
NiFi at this point. If the podling is interested in growing outside
community I'd be happy to sign on as a mentor to help that happen.

On Thu, Nov 20, 2014 at 3:32 AM, Benson Margulies <bi...@gmail.com>
wrote:

> Sean,
>
> The precedent of Accumulo is that the govt people and agencies involved are
> ready and able to have their staff collaborate openly in an Apache
> community. There's no need to contemplate bifurcation; we have this
> proposal because the management recognizes that this collaboration produces
> better stuff that solves more problems than the 'inside the tent'
> alternative.
>
> --benson
>
>
> On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > I'm really excited to see NiFi come to the incubator; it'd be a great
> > addition to the ASF.
> >
> > A few points in the proposal:
> >
> > > >== Initial Goals ==
> >
> > One of these should be to grow the community outside of the current
> niche,
> > IMHO.
> >
> > More on this below under "orphaned projects"
> >
> > > >  * Determine and establish a mechanism, possibly including a
> > > >sub-project construct, that allows for extensions to the core
> > > >application to occur at a pace that differs from the core application
> > > >itself.
> >
> > I don't think the proposal needs to include the "e.g. with sub-projects"
> > part. Just noting
> > that your goals in the incubator are to address the need to have
> different
> > release cycles
> > for core and extensions is sufficient.
> >
> >
> > > >=== Community ===
> > > >Over the past several years, NiFi has developed a strong community of
> > > >both developers and operators within the U.S. government.  We look
> > > >forward to helping grow this to a broader base of industries.
> > > >
> >
> > How much, if any, of this community do you expect to engage via the
> > customary project
> > lists once NiFi is established within the ASF? Will the project be able
> to
> > leverage this
> > established group?
> >
> >
> > > >=== Orphaned Products ===
> > > >Risk of orphaning is minimal.  The project user and developer base is
> > > >substantial, growing, and there is already extensive operational use
> > > >of NiFi.
> >
> > Given that the established base is internal to the U.S. government, I'd
> > encourage the
> > podling to consider the risk of a bifurcated project should a substantial
> > outside
> > community fail to emerge or if those internal users should fail to engage
> > with the
> > outside community.
> >
> > You cover a related issue in your "Homogenous Developers" section. But I
> > think
> > building on the "Community" section of the current state to call this out
> > as an
> > independent issue is worthwhile.
> >
> >
> > > >possible.  This environment includes widely accessible source code
> > > >repositories, published artifacts, ticket tracking, and extensive
> > > >documentation. We also encourage contributions and frequent debate and
> > > >hold regular, collaborative discussions through e-mail, chat rooms,
> > > >and in-person meet-ups.
> >
> > Do you anticipate any difficulties moving these established communication
> > mechanisms to ASF public lists?
> >
> > > >=== Documentation ===
> > > >At this time there is no NiFi documentation on the web.  However, we
> > > >have extensive documentation included within the application that
> > > >details usage of the many functions.  We will be rapidly expanding the
> > > >available documentation to cover things like installation, developer
> > > >guide, frequently asked questions, best practices, and more.  This
> > > >documentation will be posted to the NiFi wiki at apache.org.
> >
> > I love projects that start with documentation. :)
> >
> > I don't think the proposal needs to include that the documentation will
> be
> > posted
> > to the NiFi wiki, since that's an implementation detail. Just say "this
> > documentation
> > will be made available via the NiFi project's use of incubator infra."
> >
> > (I'll save detail for the eventual dev@ list, but you should strongly
> > consider not
> > using the wiki to host this documentation.)
> >
> > -Sean
> >
> > On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland <br...@cloudera.com>
> wrote:
> >
> > > Hi Joe,
> > >
> > > I know you've done a tremendous amount of work to make this happen so I
> > am
> > > extremely happy this is *finally* making it's way to the incubator!
> > >
> > > I look forward to helping in anyway I can.
> > >
> > > Cheers!
> > > Brock
> > >
> > > On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980) <
> > > chris.a.mattmann@jpl.nasa.gov> wrote:
> > >
> > > > This is *fan freakin¹ tastic* Sounds like an awesome  project and
> > > > glad to hear a relationship to Tika! Awesome to see more government
> > > > projects coming into the ASF!
> > > >
> > > > you already have a great set of mentors and I don¹t really have more
> > > > time on my plate, but really happy and will try and monitor and help
> > > > on the lists.
> > > >
> > > > Cheers!
> > > >
> > > > Chris
> > > >
> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > Chris Mattmann, Ph.D.
> > > > Chief Architect
> > > > Instrument Software and Science Data Systems Section (398)
> > > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > > Office: 168-519, Mailstop: 168-527
> > > > Email: chris.a.mattmann@nasa.gov
> > > > WWW:  http://sunset.usc.edu/~mattmann/
> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > > Adjunct Associate Professor, Computer Science Department
> > > > University of Southern California, Los Angeles, CA 90089 USA
> > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Joe Witt <jo...@gmail.com>
> > > > Reply-To: "general@incubator.apache.org" <
> general@incubator.apache.org
> > >
> > > > Date: Thursday, November 20, 2014 at 3:02 AM
> > > > To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> > > > Subject: [PROPOSAL] NiFi for Incubation
> > > >
> > > > >Hello,
> > > > >
> > > > >
> > > > >I would like to propose NiFi as an Apache Incubator Project.
> > > > >
> > > > >In addition to the copy provided below the Wiki version of the
> > > > >proposal can be found here:
> > > > >http://wiki.apache.org/incubator/NiFiProposal
> > > > >
> > > > >Thanks
> > > > >
> > > > >Joe
> > > > >
> > > > >
> > > > >= NiFi Proposal =
> > > > >
> > > > >== Abstract ==
> > > > >NiFi is a dataflow system based on the concepts of flow-based
> > > programming.
> > > > >
> > > > >== Proposal ==
> > > > >NiFi supports powerful and scalable directed graphs of data routing,
> > > > >transformation, and system mediation logic.  Some of the high-level
> > > > >capabilities and objectives of NiFi include:
> > > > >  * Web-based user interface for seamless experience between design,
> > > > >control, feedback, and monitoring of data flows
> > > > >  * Highly configurable along several dimensions of quality of
> service
> > > > >such as loss tolerant versus guaranteed delivery, low latency versus
> > > > >high throughput, and priority based queuing
> > > > >  * Fine-grained data provenance for all data received, forked,
> > > > >joined, cloned, modified, sent, and ultimately dropped as data
> reaches
> > > > >its configured end-state
> > > > >  * Component-based extension model along well defined interfaces
> > > > >enabling rapid development and effective testing
> > > > >
> > > > >== Background ==
> > > > >Reliable and effective dataflow between systems can be difficult
> > > > >whether you're running scripts on a laptop or have a massive
> > > > >distributed computing system operated by numerous teams and
> > > > >organizations.  As the volume and rate of data grows and as the
> number
> > > > >of systems, protocols, and formats increase and evolve so too does
> the
> > > > >complexity and need for greater insight and agility.  These are the
> > > > >dataflow challenges that NiFi was built to tackle.
> > > > >
> > > > >NiFi is designed in a manner consistent with the core concepts
> > > > >described in flow-based programming as originally documented by J.
> > > > >Paul Morrison in the 1970s.  This model lends itself well to visual
> > > > >diagramming, concurrency, componentization, testing, and reuse.  In
> > > > >addition to staying close to the fundamentals of flow-based
> > > > >programming, NiFi provides integration system specific features such
> > > > >as: guaranteed delivery; back pressure; ability to gracefully handle
> > > > >backlogs and data surges; and an operator interface that enables
> > > > >on-the-fly data flow generation, modification, and observation.
> > > > >
> > > > >== Rationale ==
> > > > >NiFi provides a reliable, scalable, manageable and accountable
> > > > >platform for developers and technical staff to create and evolve
> > > > >powerful data flows.  Such a system is useful in many contexts
> > > > >including large-scale enterprise integration, interaction with cloud
> > > > >services and frameworks, business to business, intra-departmental,
> and
> > > > >inter-departmental flows.  NiFi fits well within the Apache Software
> > > > >Foundation (ASF) family as it depends on numerous ASF projects and
> > > > >integrates with several others.  We also anticipate developing
> > > > >extensions for several other ASF projects such as Cassandra, Kafka,
> > > > >and Storm in the near future.
> > > > >
> > > > >== Initial Goals ==
> > > > >  * Ensure all dependencies are compliant with Apache License
> version
> > > > >2.0 and all that all code and documentation artifacts have the
> correct
> > > > >Apache licensing markings and notice.
> > > > >  * Establish a formal release process and schedule, allowing for
> > > > >dependable release cycles in a manner consistent with the Apache
> > > > >development process.
> > > > >  * Determine and establish a mechanism, possibly including a
> > > > >sub-project construct, that allows for extensions to the core
> > > > >application to occur at a pace that differs from the core
> application
> > > > >itself.
> > > > >
> > > > >== Current Status ==
> > > > >=== Meritocracy ===
> > > > >An integration platform is only as good as its ability to integrate
> > > > >systems in a reliable, timely, and repeatable manner.  The same can
> be
> > > > >said of its ability to attract talent and a variety of perspectives
> as
> > > > >integration systems by their nature are always evolving.  We will
> > > > >actively seek help and encourage promotion of influence in the
> project
> > > > >through meritocracy.
> > > > >
> > > > >=== Community ===
> > > > >Over the past several years, NiFi has developed a strong community
> of
> > > > >both developers and operators within the U.S. government.  We look
> > > > >forward to helping grow this to a broader base of industries.
> > > > >
> > > > >=== Core Developers ===
> > > > >The initial core developers are employed by the National Security
> > > > >Agency and defense contractors.  We will work to grow the community
> > > > >among a more diverse set of developers and industries.
> > > > >
> > > > >=== Alignment ===
> > > > >From its inception, NiFi was developed with an open source
> philosophy
> > > > >in mind and with the hopes of eventually being truly open sourced.
> > > > >The Apache way is consistent with the approach we have taken to
> date.
> > > > >The ASF clearly provides a mature and effective environment for
> > > > >successful development as is evident across the spectrum of
> well-known
> > > > >projects.  Further, NiFi depends on numerous ASF libraries and
> > > > >projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
> > > > >HttpClient, Jakarta and Maven.  We also anticipate extensions and
> > > > >dependencies with several more ASF projects, including Accumulo,
> Avro,
> > > > >Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
> > > > >This existing alignment with Apache and the desired community makes
> > > > >the Apache Incubator a good fit for NiFi.
> > > > >
> > > > >== Known Risks ==
> > > > >=== Orphaned Products ===
> > > > >Risk of orphaning is minimal.  The project user and developer base
> is
> > > > >substantial, growing, and there is already extensive operational use
> > > > >of NiFi.
> > > > >
> > > > >=== Inexperience with Open Source ===
> > > > >The initial committers to NiFi have limited experience with true
> open
> > > > >source software development.  However, despite the project origins
> > > > >being from closed source development we have modelled our behavior
> and
> > > > >community development on The Apache Way to the greatest extent
> > > > >possible.  This environment includes widely accessible source code
> > > > >repositories, published artifacts, ticket tracking, and extensive
> > > > >documentation. We also encourage contributions and frequent debate
> and
> > > > >hold regular, collaborative discussions through e-mail, chat rooms,
> > > > >and in-person meet-ups.  We are committed to the ideals of open
> source
> > > > >software and will eagerly seek out mentors and sponsors who can help
> > > > >us quickly come up to speed.
> > > > >
> > > > >=== Homogenous Developers ===
> > > > >The initial committers of NiFi come from a limited set of entities
> > > > >though we are committed to recruiting and developing additional
> > > > >committers from a broad spectrum of industries and backgrounds.
> > > > >
> > > > >=== Reliance on Salaried Developers ===
> > > > >We expect NiFi development to continue on salaried time and through
> > > > >volunteer time.  The initial committers are paid by their employers
> to
> > > > >contribute to this project.  We are committed to developing and
> > > > >recruiting participation from developers both salaried and
> > > > >non-salaried.
> > > > >
> > > > >=== Relationship with other Apache Projects ===
> > > > >As described in the alignment section, NiFi is already heavily
> > > > >dependent on other ASF projects and we anticipate further dependence
> > > > >and integration with new and emerging projects in the Apache family.
> > > > >
> > > > >=== An Excessive Fascination with the Apache Brand ===
> > > > >We respect the laudable Apache brand and that is certainly a factor
> in
> > > > >the decision to propose NiFi for the Apache Incubator.  We intend to
> > > > >provide a great deal of energy and capability to the ASF through
> this
> > > > >project.  We will be sensitive to and respectful of any overuse of
> the
> > > > >Apache brand and ensure our focus remains on how we benefit the
> Apache
> > > > >community.
> > > > >
> > > > >=== Documentation ===
> > > > >At this time there is no NiFi documentation on the web.  However, we
> > > > >have extensive documentation included within the application that
> > > > >details usage of the many functions.  We will be rapidly expanding
> the
> > > > >available documentation to cover things like installation, developer
> > > > >guide, frequently asked questions, best practices, and more.  This
> > > > >documentation will be posted to the NiFi wiki at apache.org.
> > > > >
> > > > >== Initial Source ==
> > > > >NiFi has been in active development since late 2006 with
> contributions
> > > > >from dozens of developers and feedback from hundreds of users and
> > > > >developers.  The core codebase is written in Java and includes
> > > > >detailed Javadocs and feature documentation.
> > > > >
> > > > >== Source and Intellectual Property Submission ==
> > > > >Previously referred to as Niagarafiles, the NiFi code and
> > > > >documentation materials will be submitted by the National Security
> > > > >Agency.  NiFi has been developed by a mix of government employees
> and
> > > > >private companies under government contract.  Material developed by
> > > > >the government employees is in the public domain and no U.S.
> copyright
> > > > >exists in works of the federal government.  For the contractor
> > > > >developed material in the initial submission, the U.S. Government
> has
> > > > >sufficient authority to open source per DFARS 252.227-7014.  NSA
> has
> > > > >submitted the Software Grant Agreement and Corporate Contributor
> > > > >License Agreement to the Apache Software Foundation.
> > > > >
> > > > >== External Dependencies ==
> > > > >We have at least one dependency on an LGPL library which we will
> > > > >promptly address.  Otherwise, we believe all current dependencies
> are
> > > > >compatible with the ASF guidelines.  Our dependency licenses come
> from
> > > > >the following license styles:  Apache v 2.0, BSD, Public Domain,
> > > > >Eclipse Public v1, MIT, CDDL v1.
> > > > >
> > > > >== Cryptography ==
> > > > >Consistent with http://www.apache.org/licenses/exports/ we believe
> > > > >NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
> > > > >cryptographic algorithms but is designed to use algorithms provided
> by
> > > > >Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
> > > > >These cryptographic algorithm providers are used to support SSL,
> > > > >SSH/SFTP, and the encryption and decryption of sensitive properties.
> > > > >In the event that it becomes necessary we will engage with
> appropriate
> > > > >Apache members to ensure we file any necessary paperwork or
> clarified
> > > > >any cryptographic export license concerns.
> > > > >
> > > > >== Required Resources ==
> > > > >=== Mailing Lists ===
> > > > >  * user@nifi.incubator.apache.org
> > > > >  * dev@nifi.incubator.apache.org
> > > > >  * private@nifi.incubator.apache.org
> > > > >  * commits@nifi.incubator.apache.org
> > > > >
> > > > >=== Source Control ===
> > > > >NiFi requests use of Git for source control
> > > > >(git://git.apache.org/nifi.git).  We request a writeable Git repo
> for
> > > > >NiFi with mirroring to be setup to Github through INFRA.  We request
> > > > >sponsor Benson Margulies (bimargulies) to assist with creating the
> > > > >INFRA ticket for this.
> > > > >
> > > > >=== Issue Tracking ===
> > > > >JIRA NiFi (NIFI)
> > > > >
> > > > >=== Initial Committers ===
> > > > >  * Brandon Devries <brandon.devries at gmail dot com>
> > > > >  * Matt Gilman <matt.c.gilman at gmail dot com>
> > > > >  * Tony Kurc <trkurc at gmail dot com>
> > > > >  * Mark Payne <markap14 at hotmail dot com>
> > > > >  * Adam Taft <adam at adamtaft dot com>
> > > > >  * Joseph Witt <joewitt at gmail dot com>
> > > > >
> > > > >=== Affiliations ===
> > > > >  * Brandon Devries (Requitest, Inc.)
> > > > >  * Matt Gilman (Raytheon)
> > > > >  * Tony Kurc (National Security Agency)
> > > > >  * Mark Payne (Sotera Defense Solutions, Inc.)
> > > > >  * Adam Taft (Requitest, Inc.)
> > > > >  * Joseph Witt (National Security Agency)
> > > > >
> > > > >== Sponsors ==
> > > > >=== Champion ===
> > > > >  * Benson Margulies (Basis Technology) <bimargulies at apache dot
> > org>
> > > > >
> > > > >=== Nominated Mentors ===
> > > > >  * Drew Farris (Booz Allen Hamilton) <drew at apache dot org>
> > > > >  * Brock Noland (Cloudera) <brock at apache dot org>
> > > > >  * Billie Rinaldi (Hortonworks) <billie at apache dot org>
> > > > >
> > > > >=== Sponsoring Entity ===
> > > > >We request the Apache Incubator to sponsor this project.
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Sean
> >
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [PROPOSAL] NiFi for Incubation

[Benson Margulies <bi...@gmail.com>]

Sean,

The precedent of Accumulo is that the govt people and agencies involved are
ready and able to have their staff collaborate openly in an Apache
community. There's no need to contemplate bifurcation; we have this
proposal because the management recognizes that this collaboration produces
better stuff that solves more problems than the 'inside the tent'
alternative.

--benson


On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey <bu...@cloudera.com> wrote:

> I'm really excited to see NiFi come to the incubator; it'd be a great
> addition to the ASF.
>
> A few points in the proposal:
>
> > >== Initial Goals ==
>
> One of these should be to grow the community outside of the current niche,
> IMHO.
>
> More on this below under "orphaned projects"
>
> > >  * Determine and establish a mechanism, possibly including a
> > >sub-project construct, that allows for extensions to the core
> > >application to occur at a pace that differs from the core application
> > >itself.
>
> I don't think the proposal needs to include the "e.g. with sub-projects"
> part. Just noting
> that your goals in the incubator are to address the need to have different
> release cycles
> for core and extensions is sufficient.
>
>
> > >=== Community ===
> > >Over the past several years, NiFi has developed a strong community of
> > >both developers and operators within the U.S. government.  We look
> > >forward to helping grow this to a broader base of industries.
> > >
>
> How much, if any, of this community do you expect to engage via the
> customary project
> lists once NiFi is established within the ASF? Will the project be able to
> leverage this
> established group?
>
>
> > >=== Orphaned Products ===
> > >Risk of orphaning is minimal.  The project user and developer base is
> > >substantial, growing, and there is already extensive operational use
> > >of NiFi.
>
> Given that the established base is internal to the U.S. government, I'd
> encourage the
> podling to consider the risk of a bifurcated project should a substantial
> outside
> community fail to emerge or if those internal users should fail to engage
> with the
> outside community.
>
> You cover a related issue in your "Homogenous Developers" section. But I
> think
> building on the "Community" section of the current state to call this out
> as an
> independent issue is worthwhile.
>
>
> > >possible.  This environment includes widely accessible source code
> > >repositories, published artifacts, ticket tracking, and extensive
> > >documentation. We also encourage contributions and frequent debate and
> > >hold regular, collaborative discussions through e-mail, chat rooms,
> > >and in-person meet-ups.
>
> Do you anticipate any difficulties moving these established communication
> mechanisms to ASF public lists?
>
> > >=== Documentation ===
> > >At this time there is no NiFi documentation on the web.  However, we
> > >have extensive documentation included within the application that
> > >details usage of the many functions.  We will be rapidly expanding the
> > >available documentation to cover things like installation, developer
> > >guide, frequently asked questions, best practices, and more.  This
> > >documentation will be posted to the NiFi wiki at apache.org.
>
> I love projects that start with documentation. :)
>
> I don't think the proposal needs to include that the documentation will be
> posted
> to the NiFi wiki, since that's an implementation detail. Just say "this
> documentation
> will be made available via the NiFi project's use of incubator infra."
>
> (I'll save detail for the eventual dev@ list, but you should strongly
> consider not
> using the wiki to host this documentation.)
>
> -Sean
>
> On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland <br...@cloudera.com> wrote:
>
> > Hi Joe,
> >
> > I know you've done a tremendous amount of work to make this happen so I
> am
> > extremely happy this is *finally* making it's way to the incubator!
> >
> > I look forward to helping in anyway I can.
> >
> > Cheers!
> > Brock
> >
> > On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980) <
> > chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> > > This is *fan freakin¹ tastic* Sounds like an awesome  project and
> > > glad to hear a relationship to Tika! Awesome to see more government
> > > projects coming into the ASF!
> > >
> > > you already have a great set of mentors and I don¹t really have more
> > > time on my plate, but really happy and will try and monitor and help
> > > on the lists.
> > >
> > > Cheers!
> > >
> > > Chris
> > >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > Chris Mattmann, Ph.D.
> > > Chief Architect
> > > Instrument Software and Science Data Systems Section (398)
> > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > Office: 168-519, Mailstop: 168-527
> > > Email: chris.a.mattmann@nasa.gov
> > > WWW:  http://sunset.usc.edu/~mattmann/
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > Adjunct Associate Professor, Computer Science Department
> > > University of Southern California, Los Angeles, CA 90089 USA
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Joe Witt <jo...@gmail.com>
> > > Reply-To: "general@incubator.apache.org" <general@incubator.apache.org
> >
> > > Date: Thursday, November 20, 2014 at 3:02 AM
> > > To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> > > Subject: [PROPOSAL] NiFi for Incubation
> > >
> > > >Hello,
> > > >
> > > >
> > > >I would like to propose NiFi as an Apache Incubator Project.
> > > >
> > > >In addition to the copy provided below the Wiki version of the
> > > >proposal can be found here:
> > > >http://wiki.apache.org/incubator/NiFiProposal
> > > >
> > > >Thanks
> > > >
> > > >Joe
> > > >
> > > >
> > > >= NiFi Proposal =
> > > >
> > > >== Abstract ==
> > > >NiFi is a dataflow system based on the concepts of flow-based
> > programming.
> > > >
> > > >== Proposal ==
> > > >NiFi supports powerful and scalable directed graphs of data routing,
> > > >transformation, and system mediation logic.  Some of the high-level
> > > >capabilities and objectives of NiFi include:
> > > >  * Web-based user interface for seamless experience between design,
> > > >control, feedback, and monitoring of data flows
> > > >  * Highly configurable along several dimensions of quality of service
> > > >such as loss tolerant versus guaranteed delivery, low latency versus
> > > >high throughput, and priority based queuing
> > > >  * Fine-grained data provenance for all data received, forked,
> > > >joined, cloned, modified, sent, and ultimately dropped as data reaches
> > > >its configured end-state
> > > >  * Component-based extension model along well defined interfaces
> > > >enabling rapid development and effective testing
> > > >
> > > >== Background ==
> > > >Reliable and effective dataflow between systems can be difficult
> > > >whether you're running scripts on a laptop or have a massive
> > > >distributed computing system operated by numerous teams and
> > > >organizations.  As the volume and rate of data grows and as the number
> > > >of systems, protocols, and formats increase and evolve so too does the
> > > >complexity and need for greater insight and agility.  These are the
> > > >dataflow challenges that NiFi was built to tackle.
> > > >
> > > >NiFi is designed in a manner consistent with the core concepts
> > > >described in flow-based programming as originally documented by J.
> > > >Paul Morrison in the 1970s.  This model lends itself well to visual
> > > >diagramming, concurrency, componentization, testing, and reuse.  In
> > > >addition to staying close to the fundamentals of flow-based
> > > >programming, NiFi provides integration system specific features such
> > > >as: guaranteed delivery; back pressure; ability to gracefully handle
> > > >backlogs and data surges; and an operator interface that enables
> > > >on-the-fly data flow generation, modification, and observation.
> > > >
> > > >== Rationale ==
> > > >NiFi provides a reliable, scalable, manageable and accountable
> > > >platform for developers and technical staff to create and evolve
> > > >powerful data flows.  Such a system is useful in many contexts
> > > >including large-scale enterprise integration, interaction with cloud
> > > >services and frameworks, business to business, intra-departmental, and
> > > >inter-departmental flows.  NiFi fits well within the Apache Software
> > > >Foundation (ASF) family as it depends on numerous ASF projects and
> > > >integrates with several others.  We also anticipate developing
> > > >extensions for several other ASF projects such as Cassandra, Kafka,
> > > >and Storm in the near future.
> > > >
> > > >== Initial Goals ==
> > > >  * Ensure all dependencies are compliant with Apache License version
> > > >2.0 and all that all code and documentation artifacts have the correct
> > > >Apache licensing markings and notice.
> > > >  * Establish a formal release process and schedule, allowing for
> > > >dependable release cycles in a manner consistent with the Apache
> > > >development process.
> > > >  * Determine and establish a mechanism, possibly including a
> > > >sub-project construct, that allows for extensions to the core
> > > >application to occur at a pace that differs from the core application
> > > >itself.
> > > >
> > > >== Current Status ==
> > > >=== Meritocracy ===
> > > >An integration platform is only as good as its ability to integrate
> > > >systems in a reliable, timely, and repeatable manner.  The same can be
> > > >said of its ability to attract talent and a variety of perspectives as
> > > >integration systems by their nature are always evolving.  We will
> > > >actively seek help and encourage promotion of influence in the project
> > > >through meritocracy.
> > > >
> > > >=== Community ===
> > > >Over the past several years, NiFi has developed a strong community of
> > > >both developers and operators within the U.S. government.  We look
> > > >forward to helping grow this to a broader base of industries.
> > > >
> > > >=== Core Developers ===
> > > >The initial core developers are employed by the National Security
> > > >Agency and defense contractors.  We will work to grow the community
> > > >among a more diverse set of developers and industries.
> > > >
> > > >=== Alignment ===
> > > >From its inception, NiFi was developed with an open source philosophy
> > > >in mind and with the hopes of eventually being truly open sourced.
> > > >The Apache way is consistent with the approach we have taken to date.
> > > >The ASF clearly provides a mature and effective environment for
> > > >successful development as is evident across the spectrum of well-known
> > > >projects.  Further, NiFi depends on numerous ASF libraries and
> > > >projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
> > > >HttpClient, Jakarta and Maven.  We also anticipate extensions and
> > > >dependencies with several more ASF projects, including Accumulo, Avro,
> > > >Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
> > > >This existing alignment with Apache and the desired community makes
> > > >the Apache Incubator a good fit for NiFi.
> > > >
> > > >== Known Risks ==
> > > >=== Orphaned Products ===
> > > >Risk of orphaning is minimal.  The project user and developer base is
> > > >substantial, growing, and there is already extensive operational use
> > > >of NiFi.
> > > >
> > > >=== Inexperience with Open Source ===
> > > >The initial committers to NiFi have limited experience with true open
> > > >source software development.  However, despite the project origins
> > > >being from closed source development we have modelled our behavior and
> > > >community development on The Apache Way to the greatest extent
> > > >possible.  This environment includes widely accessible source code
> > > >repositories, published artifacts, ticket tracking, and extensive
> > > >documentation. We also encourage contributions and frequent debate and
> > > >hold regular, collaborative discussions through e-mail, chat rooms,
> > > >and in-person meet-ups.  We are committed to the ideals of open source
> > > >software and will eagerly seek out mentors and sponsors who can help
> > > >us quickly come up to speed.
> > > >
> > > >=== Homogenous Developers ===
> > > >The initial committers of NiFi come from a limited set of entities
> > > >though we are committed to recruiting and developing additional
> > > >committers from a broad spectrum of industries and backgrounds.
> > > >
> > > >=== Reliance on Salaried Developers ===
> > > >We expect NiFi development to continue on salaried time and through
> > > >volunteer time.  The initial committers are paid by their employers to
> > > >contribute to this project.  We are committed to developing and
> > > >recruiting participation from developers both salaried and
> > > >non-salaried.
> > > >
> > > >=== Relationship with other Apache Projects ===
> > > >As described in the alignment section, NiFi is already heavily
> > > >dependent on other ASF projects and we anticipate further dependence
> > > >and integration with new and emerging projects in the Apache family.
> > > >
> > > >=== An Excessive Fascination with the Apache Brand ===
> > > >We respect the laudable Apache brand and that is certainly a factor in
> > > >the decision to propose NiFi for the Apache Incubator.  We intend to
> > > >provide a great deal of energy and capability to the ASF through this
> > > >project.  We will be sensitive to and respectful of any overuse of the
> > > >Apache brand and ensure our focus remains on how we benefit the Apache
> > > >community.
> > > >
> > > >=== Documentation ===
> > > >At this time there is no NiFi documentation on the web.  However, we
> > > >have extensive documentation included within the application that
> > > >details usage of the many functions.  We will be rapidly expanding the
> > > >available documentation to cover things like installation, developer
> > > >guide, frequently asked questions, best practices, and more.  This
> > > >documentation will be posted to the NiFi wiki at apache.org.
> > > >
> > > >== Initial Source ==
> > > >NiFi has been in active development since late 2006 with contributions
> > > >from dozens of developers and feedback from hundreds of users and
> > > >developers.  The core codebase is written in Java and includes
> > > >detailed Javadocs and feature documentation.
> > > >
> > > >== Source and Intellectual Property Submission ==
> > > >Previously referred to as Niagarafiles, the NiFi code and
> > > >documentation materials will be submitted by the National Security
> > > >Agency.  NiFi has been developed by a mix of government employees and
> > > >private companies under government contract.  Material developed by
> > > >the government employees is in the public domain and no U.S. copyright
> > > >exists in works of the federal government.  For the contractor
> > > >developed material in the initial submission, the U.S. Government has
> > > >sufficient authority to open source per DFARS 252.227-7014.  NSA has
> > > >submitted the Software Grant Agreement and Corporate Contributor
> > > >License Agreement to the Apache Software Foundation.
> > > >
> > > >== External Dependencies ==
> > > >We have at least one dependency on an LGPL library which we will
> > > >promptly address.  Otherwise, we believe all current dependencies are
> > > >compatible with the ASF guidelines.  Our dependency licenses come from
> > > >the following license styles:  Apache v 2.0, BSD, Public Domain,
> > > >Eclipse Public v1, MIT, CDDL v1.
> > > >
> > > >== Cryptography ==
> > > >Consistent with http://www.apache.org/licenses/exports/ we believe
> > > >NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
> > > >cryptographic algorithms but is designed to use algorithms provided by
> > > >Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
> > > >These cryptographic algorithm providers are used to support SSL,
> > > >SSH/SFTP, and the encryption and decryption of sensitive properties.
> > > >In the event that it becomes necessary we will engage with appropriate
> > > >Apache members to ensure we file any necessary paperwork or clarified
> > > >any cryptographic export license concerns.
> > > >
> > > >== Required Resources ==
> > > >=== Mailing Lists ===
> > > >  * user@nifi.incubator.apache.org
> > > >  * dev@nifi.incubator.apache.org
> > > >  * private@nifi.incubator.apache.org
> > > >  * commits@nifi.incubator.apache.org
> > > >
> > > >=== Source Control ===
> > > >NiFi requests use of Git for source control
> > > >(git://git.apache.org/nifi.git).  We request a writeable Git repo for
> > > >NiFi with mirroring to be setup to Github through INFRA.  We request
> > > >sponsor Benson Margulies (bimargulies) to assist with creating the
> > > >INFRA ticket for this.
> > > >
> > > >=== Issue Tracking ===
> > > >JIRA NiFi (NIFI)
> > > >
> > > >=== Initial Committers ===
> > > >  * Brandon Devries <brandon.devries at gmail dot com>
> > > >  * Matt Gilman <matt.c.gilman at gmail dot com>
> > > >  * Tony Kurc <trkurc at gmail dot com>
> > > >  * Mark Payne <markap14 at hotmail dot com>
> > > >  * Adam Taft <adam at adamtaft dot com>
> > > >  * Joseph Witt <joewitt at gmail dot com>
> > > >
> > > >=== Affiliations ===
> > > >  * Brandon Devries (Requitest, Inc.)
> > > >  * Matt Gilman (Raytheon)
> > > >  * Tony Kurc (National Security Agency)
> > > >  * Mark Payne (Sotera Defense Solutions, Inc.)
> > > >  * Adam Taft (Requitest, Inc.)
> > > >  * Joseph Witt (National Security Agency)
> > > >
> > > >== Sponsors ==
> > > >=== Champion ===
> > > >  * Benson Margulies (Basis Technology) <bimargulies at apache dot
> org>
> > > >
> > > >=== Nominated Mentors ===
> > > >  * Drew Farris (Booz Allen Hamilton) <drew at apache dot org>
> > > >  * Brock Noland (Cloudera) <brock at apache dot org>
> > > >  * Billie Rinaldi (Hortonworks) <billie at apache dot org>
> > > >
> > > >=== Sponsoring Entity ===
> > > >We request the Apache Incubator to sponsor this project.
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
> >
>
>
>
> --
> Sean
>

Re: [PROPOSAL] NiFi for Incubation

[Joe Witt <jo...@gmail.com>]

Hello

Thank you for all the feedback thus far.

Sean, Jan I,

I've adjusted the proposal for the goals, community, and documentation.

Thanks
Joe

On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey <bu...@cloudera.com> wrote:

> I'm really excited to see NiFi come to the incubator; it'd be a great
> addition to the ASF.
>
> A few points in the proposal:
>
> > >== Initial Goals ==
>
> One of these should be to grow the community outside of the current niche,
> IMHO.
>
> More on this below under "orphaned projects"
>
> > >  * Determine and establish a mechanism, possibly including a
> > >sub-project construct, that allows for extensions to the core
> > >application to occur at a pace that differs from the core application
> > >itself.
>
> I don't think the proposal needs to include the "e.g. with sub-projects"
> part. Just noting
> that your goals in the incubator are to address the need to have different
> release cycles
> for core and extensions is sufficient.
>
>
> > >=== Community ===
> > >Over the past several years, NiFi has developed a strong community of
> > >both developers and operators within the U.S. government.  We look
> > >forward to helping grow this to a broader base of industries.
> > >
>
> How much, if any, of this community do you expect to engage via the
> customary project
> lists once NiFi is established within the ASF? Will the project be able to
> leverage this
> established group?
>
>
> > >=== Orphaned Products ===
> > >Risk of orphaning is minimal.  The project user and developer base is
> > >substantial, growing, and there is already extensive operational use
> > >of NiFi.
>
> Given that the established base is internal to the U.S. government, I'd
> encourage the
> podling to consider the risk of a bifurcated project should a substantial
> outside
> community fail to emerge or if those internal users should fail to engage
> with the
> outside community.
>
> You cover a related issue in your "Homogenous Developers" section. But I
> think
> building on the "Community" section of the current state to call this out
> as an
> independent issue is worthwhile.
>
>
> > >possible.  This environment includes widely accessible source code
> > >repositories, published artifacts, ticket tracking, and extensive
> > >documentation. We also encourage contributions and frequent debate and
> > >hold regular, collaborative discussions through e-mail, chat rooms,
> > >and in-person meet-ups.
>
> Do you anticipate any difficulties moving these established communication
> mechanisms to ASF public lists?
>
> > >=== Documentation ===
> > >At this time there is no NiFi documentation on the web.  However, we
> > >have extensive documentation included within the application that
> > >details usage of the many functions.  We will be rapidly expanding the
> > >available documentation to cover things like installation, developer
> > >guide, frequently asked questions, best practices, and more.  This
> > >documentation will be posted to the NiFi wiki at apache.org.
>
> I love projects that start with documentation. :)
>
> I don't think the proposal needs to include that the documentation will be
> posted
> to the NiFi wiki, since that's an implementation detail. Just say "this
> documentation
> will be made available via the NiFi project's use of incubator infra."
>
> (I'll save detail for the eventual dev@ list, but you should strongly
> consider not
> using the wiki to host this documentation.)
>
> -Sean
>
> On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland <br...@cloudera.com> wrote:
>
> > Hi Joe,
> >
> > I know you've done a tremendous amount of work to make this happen so I
> am
> > extremely happy this is *finally* making it's way to the incubator!
> >
> > I look forward to helping in anyway I can.
> >
> > Cheers!
> > Brock
> >
> > On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980) <
> > chris.a.mattmann@jpl.nasa.gov> wrote:
> >
> > > This is *fan freakin¹ tastic* Sounds like an awesome  project and
> > > glad to hear a relationship to Tika! Awesome to see more government
> > > projects coming into the ASF!
> > >
> > > you already have a great set of mentors and I don¹t really have more
> > > time on my plate, but really happy and will try and monitor and help
> > > on the lists.
> > >
> > > Cheers!
> > >
> > > Chris
> > >
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > Chris Mattmann, Ph.D.
> > > Chief Architect
> > > Instrument Software and Science Data Systems Section (398)
> > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > > Office: 168-519, Mailstop: 168-527
> > > Email: chris.a.mattmann@nasa.gov
> > > WWW:  http://sunset.usc.edu/~mattmann/
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > > Adjunct Associate Professor, Computer Science Department
> > > University of Southern California, Los Angeles, CA 90089 USA
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >
> > >
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Joe Witt <jo...@gmail.com>
> > > Reply-To: "general@incubator.apache.org" <general@incubator.apache.org
> >
> > > Date: Thursday, November 20, 2014 at 3:02 AM
> > > To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> > > Subject: [PROPOSAL] NiFi for Incubation
> > >
> > > >Hello,
> > > >
> > > >
> > > >I would like to propose NiFi as an Apache Incubator Project.
> > > >
> > > >In addition to the copy provided below the Wiki version of the
> > > >proposal can be found here:
> > > >http://wiki.apache.org/incubator/NiFiProposal
> > > >
> > > >Thanks
> > > >
> > > >Joe
> > > >
> > > >
> > > >= NiFi Proposal =
> > > >
> > > >== Abstract ==
> > > >NiFi is a dataflow system based on the concepts of flow-based
> > programming.
> > > >
> > > >== Proposal ==
> > > >NiFi supports powerful and scalable directed graphs of data routing,
> > > >transformation, and system mediation logic.  Some of the high-level
> > > >capabilities and objectives of NiFi include:
> > > >  * Web-based user interface for seamless experience between design,
> > > >control, feedback, and monitoring of data flows
> > > >  * Highly configurable along several dimensions of quality of service
> > > >such as loss tolerant versus guaranteed delivery, low latency versus
> > > >high throughput, and priority based queuing
> > > >  * Fine-grained data provenance for all data received, forked,
> > > >joined, cloned, modified, sent, and ultimately dropped as data reaches
> > > >its configured end-state
> > > >  * Component-based extension model along well defined interfaces
> > > >enabling rapid development and effective testing
> > > >
> > > >== Background ==
> > > >Reliable and effective dataflow between systems can be difficult
> > > >whether you're running scripts on a laptop or have a massive
> > > >distributed computing system operated by numerous teams and
> > > >organizations.  As the volume and rate of data grows and as the number
> > > >of systems, protocols, and formats increase and evolve so too does the
> > > >complexity and need for greater insight and agility.  These are the
> > > >dataflow challenges that NiFi was built to tackle.
> > > >
> > > >NiFi is designed in a manner consistent with the core concepts
> > > >described in flow-based programming as originally documented by J.
> > > >Paul Morrison in the 1970s.  This model lends itself well to visual
> > > >diagramming, concurrency, componentization, testing, and reuse.  In
> > > >addition to staying close to the fundamentals of flow-based
> > > >programming, NiFi provides integration system specific features such
> > > >as: guaranteed delivery; back pressure; ability to gracefully handle
> > > >backlogs and data surges; and an operator interface that enables
> > > >on-the-fly data flow generation, modification, and observation.
> > > >
> > > >== Rationale ==
> > > >NiFi provides a reliable, scalable, manageable and accountable
> > > >platform for developers and technical staff to create and evolve
> > > >powerful data flows.  Such a system is useful in many contexts
> > > >including large-scale enterprise integration, interaction with cloud
> > > >services and frameworks, business to business, intra-departmental, and
> > > >inter-departmental flows.  NiFi fits well within the Apache Software
> > > >Foundation (ASF) family as it depends on numerous ASF projects and
> > > >integrates with several others.  We also anticipate developing
> > > >extensions for several other ASF projects such as Cassandra, Kafka,
> > > >and Storm in the near future.
> > > >
> > > >== Initial Goals ==
> > > >  * Ensure all dependencies are compliant with Apache License version
> > > >2.0 and all that all code and documentation artifacts have the correct
> > > >Apache licensing markings and notice.
> > > >  * Establish a formal release process and schedule, allowing for
> > > >dependable release cycles in a manner consistent with the Apache
> > > >development process.
> > > >  * Determine and establish a mechanism, possibly including a
> > > >sub-project construct, that allows for extensions to the core
> > > >application to occur at a pace that differs from the core application
> > > >itself.
> > > >
> > > >== Current Status ==
> > > >=== Meritocracy ===
> > > >An integration platform is only as good as its ability to integrate
> > > >systems in a reliable, timely, and repeatable manner.  The same can be
> > > >said of its ability to attract talent and a variety of perspectives as
> > > >integration systems by their nature are always evolving.  We will
> > > >actively seek help and encourage promotion of influence in the project
> > > >through meritocracy.
> > > >
> > > >=== Community ===
> > > >Over the past several years, NiFi has developed a strong community of
> > > >both developers and operators within the U.S. government.  We look
> > > >forward to helping grow this to a broader base of industries.
> > > >
> > > >=== Core Developers ===
> > > >The initial core developers are employed by the National Security
> > > >Agency and defense contractors.  We will work to grow the community
> > > >among a more diverse set of developers and industries.
> > > >
> > > >=== Alignment ===
> > > >From its inception, NiFi was developed with an open source philosophy
> > > >in mind and with the hopes of eventually being truly open sourced.
> > > >The Apache way is consistent with the approach we have taken to date.
> > > >The ASF clearly provides a mature and effective environment for
> > > >successful development as is evident across the spectrum of well-known
> > > >projects.  Further, NiFi depends on numerous ASF libraries and
> > > >projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
> > > >HttpClient, Jakarta and Maven.  We also anticipate extensions and
> > > >dependencies with several more ASF projects, including Accumulo, Avro,
> > > >Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
> > > >This existing alignment with Apache and the desired community makes
> > > >the Apache Incubator a good fit for NiFi.
> > > >
> > > >== Known Risks ==
> > > >=== Orphaned Products ===
> > > >Risk of orphaning is minimal.  The project user and developer base is
> > > >substantial, growing, and there is already extensive operational use
> > > >of NiFi.
> > > >
> > > >=== Inexperience with Open Source ===
> > > >The initial committers to NiFi have limited experience with true open
> > > >source software development.  However, despite the project origins
> > > >being from closed source development we have modelled our behavior and
> > > >community development on The Apache Way to the greatest extent
> > > >possible.  This environment includes widely accessible source code
> > > >repositories, published artifacts, ticket tracking, and extensive
> > > >documentation. We also encourage contributions and frequent debate and
> > > >hold regular, collaborative discussions through e-mail, chat rooms,
> > > >and in-person meet-ups.  We are committed to the ideals of open source
> > > >software and will eagerly seek out mentors and sponsors who can help
> > > >us quickly come up to speed.
> > > >
> > > >=== Homogenous Developers ===
> > > >The initial committers of NiFi come from a limited set of entities
> > > >though we are committed to recruiting and developing additional
> > > >committers from a broad spectrum of industries and backgrounds.
> > > >
> > > >=== Reliance on Salaried Developers ===
> > > >We expect NiFi development to continue on salaried time and through
> > > >volunteer time.  The initial committers are paid by their employers to
> > > >contribute to this project.  We are committed to developing and
> > > >recruiting participation from developers both salaried and
> > > >non-salaried.
> > > >
> > > >=== Relationship with other Apache Projects ===
> > > >As described in the alignment section, NiFi is already heavily
> > > >dependent on other ASF projects and we anticipate further dependence
> > > >and integration with new and emerging projects in the Apache family.
> > > >
> > > >=== An Excessive Fascination with the Apache Brand ===
> > > >We respect the laudable Apache brand and that is certainly a factor in
> > > >the decision to propose NiFi for the Apache Incubator.  We intend to
> > > >provide a great deal of energy and capability to the ASF through this
> > > >project.  We will be sensitive to and respectful of any overuse of the
> > > >Apache brand and ensure our focus remains on how we benefit the Apache
> > > >community.
> > > >
> > > >=== Documentation ===
> > > >At this time there is no NiFi documentation on the web.  However, we
> > > >have extensive documentation included within the application that
> > > >details usage of the many functions.  We will be rapidly expanding the
> > > >available documentation to cover things like installation, developer
> > > >guide, frequently asked questions, best practices, and more.  This
> > > >documentation will be posted to the NiFi wiki at apache.org.
> > > >
> > > >== Initial Source ==
> > > >NiFi has been in active development since late 2006 with contributions
> > > >from dozens of developers and feedback from hundreds of users and
> > > >developers.  The core codebase is written in Java and includes
> > > >detailed Javadocs and feature documentation.
> > > >
> > > >== Source and Intellectual Property Submission ==
> > > >Previously referred to as Niagarafiles, the NiFi code and
> > > >documentation materials will be submitted by the National Security
> > > >Agency.  NiFi has been developed by a mix of government employees and
> > > >private companies under government contract.  Material developed by
> > > >the government employees is in the public domain and no U.S. copyright
> > > >exists in works of the federal government.  For the contractor
> > > >developed material in the initial submission, the U.S. Government has
> > > >sufficient authority to open source per DFARS 252.227-7014.  NSA has
> > > >submitted the Software Grant Agreement and Corporate Contributor
> > > >License Agreement to the Apache Software Foundation.
> > > >
> > > >== External Dependencies ==
> > > >We have at least one dependency on an LGPL library which we will
> > > >promptly address.  Otherwise, we believe all current dependencies are
> > > >compatible with the ASF guidelines.  Our dependency licenses come from
> > > >the following license styles:  Apache v 2.0, BSD, Public Domain,
> > > >Eclipse Public v1, MIT, CDDL v1.
> > > >
> > > >== Cryptography ==
> > > >Consistent with http://www.apache.org/licenses/exports/ we believe
> > > >NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
> > > >cryptographic algorithms but is designed to use algorithms provided by
> > > >Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
> > > >These cryptographic algorithm providers are used to support SSL,
> > > >SSH/SFTP, and the encryption and decryption of sensitive properties.
> > > >In the event that it becomes necessary we will engage with appropriate
> > > >Apache members to ensure we file any necessary paperwork or clarified
> > > >any cryptographic export license concerns.
> > > >
> > > >== Required Resources ==
> > > >=== Mailing Lists ===
> > > >  * user@nifi.incubator.apache.org
> > > >  * dev@nifi.incubator.apache.org
> > > >  * private@nifi.incubator.apache.org
> > > >  * commits@nifi.incubator.apache.org
> > > >
> > > >=== Source Control ===
> > > >NiFi requests use of Git for source control
> > > >(git://git.apache.org/nifi.git).  We request a writeable Git repo for
> > > >NiFi with mirroring to be setup to Github through INFRA.  We request
> > > >sponsor Benson Margulies (bimargulies) to assist with creating the
> > > >INFRA ticket for this.
> > > >
> > > >=== Issue Tracking ===
> > > >JIRA NiFi (NIFI)
> > > >
> > > >=== Initial Committers ===
> > > >  * Brandon Devries <brandon.devries at gmail dot com>
> > > >  * Matt Gilman <matt.c.gilman at gmail dot com>
> > > >  * Tony Kurc <trkurc at gmail dot com>
> > > >  * Mark Payne <markap14 at hotmail dot com>
> > > >  * Adam Taft <adam at adamtaft dot com>
> > > >  * Joseph Witt <joewitt at gmail dot com>
> > > >
> > > >=== Affiliations ===
> > > >  * Brandon Devries (Requitest, Inc.)
> > > >  * Matt Gilman (Raytheon)
> > > >  * Tony Kurc (National Security Agency)
> > > >  * Mark Payne (Sotera Defense Solutions, Inc.)
> > > >  * Adam Taft (Requitest, Inc.)
> > > >  * Joseph Witt (National Security Agency)
> > > >
> > > >== Sponsors ==
> > > >=== Champion ===
> > > >  * Benson Margulies (Basis Technology) <bimargulies at apache dot
> org>
> > > >
> > > >=== Nominated Mentors ===
> > > >  * Drew Farris (Booz Allen Hamilton) <drew at apache dot org>
> > > >  * Brock Noland (Cloudera) <brock at apache dot org>
> > > >  * Billie Rinaldi (Hortonworks) <billie at apache dot org>
> > > >
> > > >=== Sponsoring Entity ===
> > > >We request the Apache Incubator to sponsor this project.
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
> >
>
>
>
> --
> Sean
>

Re: [PROPOSAL] NiFi for Incubation

[Sean Busbey <bu...@cloudera.com>]

I'm really excited to see NiFi come to the incubator; it'd be a great
addition to the ASF.

A few points in the proposal:

> >== Initial Goals ==

One of these should be to grow the community outside of the current niche,
IMHO.

More on this below under "orphaned projects"

> >  * Determine and establish a mechanism, possibly including a
> >sub-project construct, that allows for extensions to the core
> >application to occur at a pace that differs from the core application
> >itself.

I don't think the proposal needs to include the "e.g. with sub-projects"
part. Just noting
that your goals in the incubator are to address the need to have different
release cycles
for core and extensions is sufficient.


> >=== Community ===
> >Over the past several years, NiFi has developed a strong community of
> >both developers and operators within the U.S. government.  We look
> >forward to helping grow this to a broader base of industries.
> >

How much, if any, of this community do you expect to engage via the
customary project
lists once NiFi is established within the ASF? Will the project be able to
leverage this
established group?


> >=== Orphaned Products ===
> >Risk of orphaning is minimal.  The project user and developer base is
> >substantial, growing, and there is already extensive operational use
> >of NiFi.

Given that the established base is internal to the U.S. government, I'd
encourage the
podling to consider the risk of a bifurcated project should a substantial
outside
community fail to emerge or if those internal users should fail to engage
with the
outside community.

You cover a related issue in your "Homogenous Developers" section. But I
think
building on the "Community" section of the current state to call this out
as an
independent issue is worthwhile.


> >possible.  This environment includes widely accessible source code
> >repositories, published artifacts, ticket tracking, and extensive
> >documentation. We also encourage contributions and frequent debate and
> >hold regular, collaborative discussions through e-mail, chat rooms,
> >and in-person meet-ups.

Do you anticipate any difficulties moving these established communication
mechanisms to ASF public lists?

> >=== Documentation ===
> >At this time there is no NiFi documentation on the web.  However, we
> >have extensive documentation included within the application that
> >details usage of the many functions.  We will be rapidly expanding the
> >available documentation to cover things like installation, developer
> >guide, frequently asked questions, best practices, and more.  This
> >documentation will be posted to the NiFi wiki at apache.org.

I love projects that start with documentation. :)

I don't think the proposal needs to include that the documentation will be
posted
to the NiFi wiki, since that's an implementation detail. Just say "this
documentation
will be made available via the NiFi project's use of incubator infra."

(I'll save detail for the eventual dev@ list, but you should strongly
consider not
using the wiki to host this documentation.)

-Sean

On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland <br...@cloudera.com> wrote:

> Hi Joe,
>
> I know you've done a tremendous amount of work to make this happen so I am
> extremely happy this is *finally* making it's way to the incubator!
>
> I look forward to helping in anyway I can.
>
> Cheers!
> Brock
>
> On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > This is *fan freakin¹ tastic* Sounds like an awesome  project and
> > glad to hear a relationship to Tika! Awesome to see more government
> > projects coming into the ASF!
> >
> > you already have a great set of mentors and I don¹t really have more
> > time on my plate, but really happy and will try and monitor and help
> > on the lists.
> >
> > Cheers!
> >
> > Chris
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Chief Architect
> > Instrument Software and Science Data Systems Section (398)
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 168-519, Mailstop: 168-527
> > Email: chris.a.mattmann@nasa.gov
> > WWW:  http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Associate Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Joe Witt <jo...@gmail.com>
> > Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> > Date: Thursday, November 20, 2014 at 3:02 AM
> > To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> > Subject: [PROPOSAL] NiFi for Incubation
> >
> > >Hello,
> > >
> > >
> > >I would like to propose NiFi as an Apache Incubator Project.
> > >
> > >In addition to the copy provided below the Wiki version of the
> > >proposal can be found here:
> > >http://wiki.apache.org/incubator/NiFiProposal
> > >
> > >Thanks
> > >
> > >Joe
> > >
> > >
> > >= NiFi Proposal =
> > >
> > >== Abstract ==
> > >NiFi is a dataflow system based on the concepts of flow-based
> programming.
> > >
> > >== Proposal ==
> > >NiFi supports powerful and scalable directed graphs of data routing,
> > >transformation, and system mediation logic.  Some of the high-level
> > >capabilities and objectives of NiFi include:
> > >  * Web-based user interface for seamless experience between design,
> > >control, feedback, and monitoring of data flows
> > >  * Highly configurable along several dimensions of quality of service
> > >such as loss tolerant versus guaranteed delivery, low latency versus
> > >high throughput, and priority based queuing
> > >  * Fine-grained data provenance for all data received, forked,
> > >joined, cloned, modified, sent, and ultimately dropped as data reaches
> > >its configured end-state
> > >  * Component-based extension model along well defined interfaces
> > >enabling rapid development and effective testing
> > >
> > >== Background ==
> > >Reliable and effective dataflow between systems can be difficult
> > >whether you're running scripts on a laptop or have a massive
> > >distributed computing system operated by numerous teams and
> > >organizations.  As the volume and rate of data grows and as the number
> > >of systems, protocols, and formats increase and evolve so too does the
> > >complexity and need for greater insight and agility.  These are the
> > >dataflow challenges that NiFi was built to tackle.
> > >
> > >NiFi is designed in a manner consistent with the core concepts
> > >described in flow-based programming as originally documented by J.
> > >Paul Morrison in the 1970s.  This model lends itself well to visual
> > >diagramming, concurrency, componentization, testing, and reuse.  In
> > >addition to staying close to the fundamentals of flow-based
> > >programming, NiFi provides integration system specific features such
> > >as: guaranteed delivery; back pressure; ability to gracefully handle
> > >backlogs and data surges; and an operator interface that enables
> > >on-the-fly data flow generation, modification, and observation.
> > >
> > >== Rationale ==
> > >NiFi provides a reliable, scalable, manageable and accountable
> > >platform for developers and technical staff to create and evolve
> > >powerful data flows.  Such a system is useful in many contexts
> > >including large-scale enterprise integration, interaction with cloud
> > >services and frameworks, business to business, intra-departmental, and
> > >inter-departmental flows.  NiFi fits well within the Apache Software
> > >Foundation (ASF) family as it depends on numerous ASF projects and
> > >integrates with several others.  We also anticipate developing
> > >extensions for several other ASF projects such as Cassandra, Kafka,
> > >and Storm in the near future.
> > >
> > >== Initial Goals ==
> > >  * Ensure all dependencies are compliant with Apache License version
> > >2.0 and all that all code and documentation artifacts have the correct
> > >Apache licensing markings and notice.
> > >  * Establish a formal release process and schedule, allowing for
> > >dependable release cycles in a manner consistent with the Apache
> > >development process.
> > >  * Determine and establish a mechanism, possibly including a
> > >sub-project construct, that allows for extensions to the core
> > >application to occur at a pace that differs from the core application
> > >itself.
> > >
> > >== Current Status ==
> > >=== Meritocracy ===
> > >An integration platform is only as good as its ability to integrate
> > >systems in a reliable, timely, and repeatable manner.  The same can be
> > >said of its ability to attract talent and a variety of perspectives as
> > >integration systems by their nature are always evolving.  We will
> > >actively seek help and encourage promotion of influence in the project
> > >through meritocracy.
> > >
> > >=== Community ===
> > >Over the past several years, NiFi has developed a strong community of
> > >both developers and operators within the U.S. government.  We look
> > >forward to helping grow this to a broader base of industries.
> > >
> > >=== Core Developers ===
> > >The initial core developers are employed by the National Security
> > >Agency and defense contractors.  We will work to grow the community
> > >among a more diverse set of developers and industries.
> > >
> > >=== Alignment ===
> > >From its inception, NiFi was developed with an open source philosophy
> > >in mind and with the hopes of eventually being truly open sourced.
> > >The Apache way is consistent with the approach we have taken to date.
> > >The ASF clearly provides a mature and effective environment for
> > >successful development as is evident across the spectrum of well-known
> > >projects.  Further, NiFi depends on numerous ASF libraries and
> > >projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
> > >HttpClient, Jakarta and Maven.  We also anticipate extensions and
> > >dependencies with several more ASF projects, including Accumulo, Avro,
> > >Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
> > >This existing alignment with Apache and the desired community makes
> > >the Apache Incubator a good fit for NiFi.
> > >
> > >== Known Risks ==
> > >=== Orphaned Products ===
> > >Risk of orphaning is minimal.  The project user and developer base is
> > >substantial, growing, and there is already extensive operational use
> > >of NiFi.
> > >
> > >=== Inexperience with Open Source ===
> > >The initial committers to NiFi have limited experience with true open
> > >source software development.  However, despite the project origins
> > >being from closed source development we have modelled our behavior and
> > >community development on The Apache Way to the greatest extent
> > >possible.  This environment includes widely accessible source code
> > >repositories, published artifacts, ticket tracking, and extensive
> > >documentation. We also encourage contributions and frequent debate and
> > >hold regular, collaborative discussions through e-mail, chat rooms,
> > >and in-person meet-ups.  We are committed to the ideals of open source
> > >software and will eagerly seek out mentors and sponsors who can help
> > >us quickly come up to speed.
> > >
> > >=== Homogenous Developers ===
> > >The initial committers of NiFi come from a limited set of entities
> > >though we are committed to recruiting and developing additional
> > >committers from a broad spectrum of industries and backgrounds.
> > >
> > >=== Reliance on Salaried Developers ===
> > >We expect NiFi development to continue on salaried time and through
> > >volunteer time.  The initial committers are paid by their employers to
> > >contribute to this project.  We are committed to developing and
> > >recruiting participation from developers both salaried and
> > >non-salaried.
> > >
> > >=== Relationship with other Apache Projects ===
> > >As described in the alignment section, NiFi is already heavily
> > >dependent on other ASF projects and we anticipate further dependence
> > >and integration with new and emerging projects in the Apache family.
> > >
> > >=== An Excessive Fascination with the Apache Brand ===
> > >We respect the laudable Apache brand and that is certainly a factor in
> > >the decision to propose NiFi for the Apache Incubator.  We intend to
> > >provide a great deal of energy and capability to the ASF through this
> > >project.  We will be sensitive to and respectful of any overuse of the
> > >Apache brand and ensure our focus remains on how we benefit the Apache
> > >community.
> > >
> > >=== Documentation ===
> > >At this time there is no NiFi documentation on the web.  However, we
> > >have extensive documentation included within the application that
> > >details usage of the many functions.  We will be rapidly expanding the
> > >available documentation to cover things like installation, developer
> > >guide, frequently asked questions, best practices, and more.  This
> > >documentation will be posted to the NiFi wiki at apache.org.
> > >
> > >== Initial Source ==
> > >NiFi has been in active development since late 2006 with contributions
> > >from dozens of developers and feedback from hundreds of users and
> > >developers.  The core codebase is written in Java and includes
> > >detailed Javadocs and feature documentation.
> > >
> > >== Source and Intellectual Property Submission ==
> > >Previously referred to as Niagarafiles, the NiFi code and
> > >documentation materials will be submitted by the National Security
> > >Agency.  NiFi has been developed by a mix of government employees and
> > >private companies under government contract.  Material developed by
> > >the government employees is in the public domain and no U.S. copyright
> > >exists in works of the federal government.  For the contractor
> > >developed material in the initial submission, the U.S. Government has
> > >sufficient authority to open source per DFARS 252.227-7014.  NSA has
> > >submitted the Software Grant Agreement and Corporate Contributor
> > >License Agreement to the Apache Software Foundation.
> > >
> > >== External Dependencies ==
> > >We have at least one dependency on an LGPL library which we will
> > >promptly address.  Otherwise, we believe all current dependencies are
> > >compatible with the ASF guidelines.  Our dependency licenses come from
> > >the following license styles:  Apache v 2.0, BSD, Public Domain,
> > >Eclipse Public v1, MIT, CDDL v1.
> > >
> > >== Cryptography ==
> > >Consistent with http://www.apache.org/licenses/exports/ we believe
> > >NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
> > >cryptographic algorithms but is designed to use algorithms provided by
> > >Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
> > >These cryptographic algorithm providers are used to support SSL,
> > >SSH/SFTP, and the encryption and decryption of sensitive properties.
> > >In the event that it becomes necessary we will engage with appropriate
> > >Apache members to ensure we file any necessary paperwork or clarified
> > >any cryptographic export license concerns.
> > >
> > >== Required Resources ==
> > >=== Mailing Lists ===
> > >  * user@nifi.incubator.apache.org
> > >  * dev@nifi.incubator.apache.org
> > >  * private@nifi.incubator.apache.org
> > >  * commits@nifi.incubator.apache.org
> > >
> > >=== Source Control ===
> > >NiFi requests use of Git for source control
> > >(git://git.apache.org/nifi.git).  We request a writeable Git repo for
> > >NiFi with mirroring to be setup to Github through INFRA.  We request
> > >sponsor Benson Margulies (bimargulies) to assist with creating the
> > >INFRA ticket for this.
> > >
> > >=== Issue Tracking ===
> > >JIRA NiFi (NIFI)
> > >
> > >=== Initial Committers ===
> > >  * Brandon Devries <brandon.devries at gmail dot com>
> > >  * Matt Gilman <matt.c.gilman at gmail dot com>
> > >  * Tony Kurc <trkurc at gmail dot com>
> > >  * Mark Payne <markap14 at hotmail dot com>
> > >  * Adam Taft <adam at adamtaft dot com>
> > >  * Joseph Witt <joewitt at gmail dot com>
> > >
> > >=== Affiliations ===
> > >  * Brandon Devries (Requitest, Inc.)
> > >  * Matt Gilman (Raytheon)
> > >  * Tony Kurc (National Security Agency)
> > >  * Mark Payne (Sotera Defense Solutions, Inc.)
> > >  * Adam Taft (Requitest, Inc.)
> > >  * Joseph Witt (National Security Agency)
> > >
> > >== Sponsors ==
> > >=== Champion ===
> > >  * Benson Margulies (Basis Technology) <bimargulies at apache dot org>
> > >
> > >=== Nominated Mentors ===
> > >  * Drew Farris (Booz Allen Hamilton) <drew at apache dot org>
> > >  * Brock Noland (Cloudera) <brock at apache dot org>
> > >  * Billie Rinaldi (Hortonworks) <billie at apache dot org>
> > >
> > >=== Sponsoring Entity ===
> > >We request the Apache Incubator to sponsor this project.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>



-- 
Sean

Re: [PROPOSAL] NiFi for Incubation

[Brock Noland <br...@cloudera.com>]

Hi Joe,

I know you've done a tremendous amount of work to make this happen so I am
extremely happy this is *finally* making it's way to the incubator!

I look forward to helping in anyway I can.

Cheers!
Brock

On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:

> This is *fan freakin¹ tastic* Sounds like an awesome  project and
> glad to hear a relationship to Tika! Awesome to see more government
> projects coming into the ASF!
>
> you already have a great set of mentors and I don¹t really have more
> time on my plate, but really happy and will try and monitor and help
> on the lists.
>
> Cheers!
>
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
>
>
> -----Original Message-----
> From: Joe Witt <jo...@gmail.com>
> Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> Date: Thursday, November 20, 2014 at 3:02 AM
> To: "general@incubator.apache.org" <ge...@incubator.apache.org>
> Subject: [PROPOSAL] NiFi for Incubation
>
> >Hello,
> >
> >
> >I would like to propose NiFi as an Apache Incubator Project.
> >
> >In addition to the copy provided below the Wiki version of the
> >proposal can be found here:
> >http://wiki.apache.org/incubator/NiFiProposal
> >
> >Thanks
> >
> >Joe
> >
> >
> >= NiFi Proposal =
> >
> >== Abstract ==
> >NiFi is a dataflow system based on the concepts of flow-based programming.
> >
> >== Proposal ==
> >NiFi supports powerful and scalable directed graphs of data routing,
> >transformation, and system mediation logic.  Some of the high-level
> >capabilities and objectives of NiFi include:
> >  * Web-based user interface for seamless experience between design,
> >control, feedback, and monitoring of data flows
> >  * Highly configurable along several dimensions of quality of service
> >such as loss tolerant versus guaranteed delivery, low latency versus
> >high throughput, and priority based queuing
> >  * Fine-grained data provenance for all data received, forked,
> >joined, cloned, modified, sent, and ultimately dropped as data reaches
> >its configured end-state
> >  * Component-based extension model along well defined interfaces
> >enabling rapid development and effective testing
> >
> >== Background ==
> >Reliable and effective dataflow between systems can be difficult
> >whether you're running scripts on a laptop or have a massive
> >distributed computing system operated by numerous teams and
> >organizations.  As the volume and rate of data grows and as the number
> >of systems, protocols, and formats increase and evolve so too does the
> >complexity and need for greater insight and agility.  These are the
> >dataflow challenges that NiFi was built to tackle.
> >
> >NiFi is designed in a manner consistent with the core concepts
> >described in flow-based programming as originally documented by J.
> >Paul Morrison in the 1970s.  This model lends itself well to visual
> >diagramming, concurrency, componentization, testing, and reuse.  In
> >addition to staying close to the fundamentals of flow-based
> >programming, NiFi provides integration system specific features such
> >as: guaranteed delivery; back pressure; ability to gracefully handle
> >backlogs and data surges; and an operator interface that enables
> >on-the-fly data flow generation, modification, and observation.
> >
> >== Rationale ==
> >NiFi provides a reliable, scalable, manageable and accountable
> >platform for developers and technical staff to create and evolve
> >powerful data flows.  Such a system is useful in many contexts
> >including large-scale enterprise integration, interaction with cloud
> >services and frameworks, business to business, intra-departmental, and
> >inter-departmental flows.  NiFi fits well within the Apache Software
> >Foundation (ASF) family as it depends on numerous ASF projects and
> >integrates with several others.  We also anticipate developing
> >extensions for several other ASF projects such as Cassandra, Kafka,
> >and Storm in the near future.
> >
> >== Initial Goals ==
> >  * Ensure all dependencies are compliant with Apache License version
> >2.0 and all that all code and documentation artifacts have the correct
> >Apache licensing markings and notice.
> >  * Establish a formal release process and schedule, allowing for
> >dependable release cycles in a manner consistent with the Apache
> >development process.
> >  * Determine and establish a mechanism, possibly including a
> >sub-project construct, that allows for extensions to the core
> >application to occur at a pace that differs from the core application
> >itself.
> >
> >== Current Status ==
> >=== Meritocracy ===
> >An integration platform is only as good as its ability to integrate
> >systems in a reliable, timely, and repeatable manner.  The same can be
> >said of its ability to attract talent and a variety of perspectives as
> >integration systems by their nature are always evolving.  We will
> >actively seek help and encourage promotion of influence in the project
> >through meritocracy.
> >
> >=== Community ===
> >Over the past several years, NiFi has developed a strong community of
> >both developers and operators within the U.S. government.  We look
> >forward to helping grow this to a broader base of industries.
> >
> >=== Core Developers ===
> >The initial core developers are employed by the National Security
> >Agency and defense contractors.  We will work to grow the community
> >among a more diverse set of developers and industries.
> >
> >=== Alignment ===
> >From its inception, NiFi was developed with an open source philosophy
> >in mind and with the hopes of eventually being truly open sourced.
> >The Apache way is consistent with the approach we have taken to date.
> >The ASF clearly provides a mature and effective environment for
> >successful development as is evident across the spectrum of well-known
> >projects.  Further, NiFi depends on numerous ASF libraries and
> >projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
> >HttpClient, Jakarta and Maven.  We also anticipate extensions and
> >dependencies with several more ASF projects, including Accumulo, Avro,
> >Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
> >This existing alignment with Apache and the desired community makes
> >the Apache Incubator a good fit for NiFi.
> >
> >== Known Risks ==
> >=== Orphaned Products ===
> >Risk of orphaning is minimal.  The project user and developer base is
> >substantial, growing, and there is already extensive operational use
> >of NiFi.
> >
> >=== Inexperience with Open Source ===
> >The initial committers to NiFi have limited experience with true open
> >source software development.  However, despite the project origins
> >being from closed source development we have modelled our behavior and
> >community development on The Apache Way to the greatest extent
> >possible.  This environment includes widely accessible source code
> >repositories, published artifacts, ticket tracking, and extensive
> >documentation. We also encourage contributions and frequent debate and
> >hold regular, collaborative discussions through e-mail, chat rooms,
> >and in-person meet-ups.  We are committed to the ideals of open source
> >software and will eagerly seek out mentors and sponsors who can help
> >us quickly come up to speed.
> >
> >=== Homogenous Developers ===
> >The initial committers of NiFi come from a limited set of entities
> >though we are committed to recruiting and developing additional
> >committers from a broad spectrum of industries and backgrounds.
> >
> >=== Reliance on Salaried Developers ===
> >We expect NiFi development to continue on salaried time and through
> >volunteer time.  The initial committers are paid by their employers to
> >contribute to this project.  We are committed to developing and
> >recruiting participation from developers both salaried and
> >non-salaried.
> >
> >=== Relationship with other Apache Projects ===
> >As described in the alignment section, NiFi is already heavily
> >dependent on other ASF projects and we anticipate further dependence
> >and integration with new and emerging projects in the Apache family.
> >
> >=== An Excessive Fascination with the Apache Brand ===
> >We respect the laudable Apache brand and that is certainly a factor in
> >the decision to propose NiFi for the Apache Incubator.  We intend to
> >provide a great deal of energy and capability to the ASF through this
> >project.  We will be sensitive to and respectful of any overuse of the
> >Apache brand and ensure our focus remains on how we benefit the Apache
> >community.
> >
> >=== Documentation ===
> >At this time there is no NiFi documentation on the web.  However, we
> >have extensive documentation included within the application that
> >details usage of the many functions.  We will be rapidly expanding the
> >available documentation to cover things like installation, developer
> >guide, frequently asked questions, best practices, and more.  This
> >documentation will be posted to the NiFi wiki at apache.org.
> >
> >== Initial Source ==
> >NiFi has been in active development since late 2006 with contributions
> >from dozens of developers and feedback from hundreds of users and
> >developers.  The core codebase is written in Java and includes
> >detailed Javadocs and feature documentation.
> >
> >== Source and Intellectual Property Submission ==
> >Previously referred to as Niagarafiles, the NiFi code and
> >documentation materials will be submitted by the National Security
> >Agency.  NiFi has been developed by a mix of government employees and
> >private companies under government contract.  Material developed by
> >the government employees is in the public domain and no U.S. copyright
> >exists in works of the federal government.  For the contractor
> >developed material in the initial submission, the U.S. Government has
> >sufficient authority to open source per DFARS 252.227-7014.  NSA has
> >submitted the Software Grant Agreement and Corporate Contributor
> >License Agreement to the Apache Software Foundation.
> >
> >== External Dependencies ==
> >We have at least one dependency on an LGPL library which we will
> >promptly address.  Otherwise, we believe all current dependencies are
> >compatible with the ASF guidelines.  Our dependency licenses come from
> >the following license styles:  Apache v 2.0, BSD, Public Domain,
> >Eclipse Public v1, MIT, CDDL v1.
> >
> >== Cryptography ==
> >Consistent with http://www.apache.org/licenses/exports/ we believe
> >NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
> >cryptographic algorithms but is designed to use algorithms provided by
> >Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
> >These cryptographic algorithm providers are used to support SSL,
> >SSH/SFTP, and the encryption and decryption of sensitive properties.
> >In the event that it becomes necessary we will engage with appropriate
> >Apache members to ensure we file any necessary paperwork or clarified
> >any cryptographic export license concerns.
> >
> >== Required Resources ==
> >=== Mailing Lists ===
> >  * user@nifi.incubator.apache.org
> >  * dev@nifi.incubator.apache.org
> >  * private@nifi.incubator.apache.org
> >  * commits@nifi.incubator.apache.org
> >
> >=== Source Control ===
> >NiFi requests use of Git for source control
> >(git://git.apache.org/nifi.git).  We request a writeable Git repo for
> >NiFi with mirroring to be setup to Github through INFRA.  We request
> >sponsor Benson Margulies (bimargulies) to assist with creating the
> >INFRA ticket for this.
> >
> >=== Issue Tracking ===
> >JIRA NiFi (NIFI)
> >
> >=== Initial Committers ===
> >  * Brandon Devries <brandon.devries at gmail dot com>
> >  * Matt Gilman <matt.c.gilman at gmail dot com>
> >  * Tony Kurc <trkurc at gmail dot com>
> >  * Mark Payne <markap14 at hotmail dot com>
> >  * Adam Taft <adam at adamtaft dot com>
> >  * Joseph Witt <joewitt at gmail dot com>
> >
> >=== Affiliations ===
> >  * Brandon Devries (Requitest, Inc.)
> >  * Matt Gilman (Raytheon)
> >  * Tony Kurc (National Security Agency)
> >  * Mark Payne (Sotera Defense Solutions, Inc.)
> >  * Adam Taft (Requitest, Inc.)
> >  * Joseph Witt (National Security Agency)
> >
> >== Sponsors ==
> >=== Champion ===
> >  * Benson Margulies (Basis Technology) <bimargulies at apache dot org>
> >
> >=== Nominated Mentors ===
> >  * Drew Farris (Booz Allen Hamilton) <drew at apache dot org>
> >  * Brock Noland (Cloudera) <brock at apache dot org>
> >  * Billie Rinaldi (Hortonworks) <billie at apache dot org>
> >
> >=== Sponsoring Entity ===
> >We request the Apache Incubator to sponsor this project.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [PROPOSAL] NiFi for Incubation

["Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>]

This is *fan freakin¹ tastic* Sounds like an awesome  project and
glad to hear a relationship to Tika! Awesome to see more government
projects coming into the ASF!

you already have a great set of mentors and I don¹t really have more
time on my plate, but really happy and will try and monitor and help
on the lists.

Cheers!

Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++






-----Original Message-----
From: Joe Witt <jo...@gmail.com>
Reply-To: "general@incubator.apache.org" <ge...@incubator.apache.org>
Date: Thursday, November 20, 2014 at 3:02 AM
To: "general@incubator.apache.org" <ge...@incubator.apache.org>
Subject: [PROPOSAL] NiFi for Incubation

>Hello,
>
>
>I would like to propose NiFi as an Apache Incubator Project.
>
>In addition to the copy provided below the Wiki version of the
>proposal can be found here:
>http://wiki.apache.org/incubator/NiFiProposal
>
>Thanks
>
>Joe
>
>
>= NiFi Proposal =
>
>== Abstract ==
>NiFi is a dataflow system based on the concepts of flow-based programming.
>
>== Proposal ==
>NiFi supports powerful and scalable directed graphs of data routing,
>transformation, and system mediation logic.  Some of the high-level
>capabilities and objectives of NiFi include:
>  * Web-based user interface for seamless experience between design,
>control, feedback, and monitoring of data flows
>  * Highly configurable along several dimensions of quality of service
>such as loss tolerant versus guaranteed delivery, low latency versus
>high throughput, and priority based queuing
>  * Fine-grained data provenance for all data received, forked,
>joined, cloned, modified, sent, and ultimately dropped as data reaches
>its configured end-state
>  * Component-based extension model along well defined interfaces
>enabling rapid development and effective testing
>
>== Background ==
>Reliable and effective dataflow between systems can be difficult
>whether you're running scripts on a laptop or have a massive
>distributed computing system operated by numerous teams and
>organizations.  As the volume and rate of data grows and as the number
>of systems, protocols, and formats increase and evolve so too does the
>complexity and need for greater insight and agility.  These are the
>dataflow challenges that NiFi was built to tackle.
>
>NiFi is designed in a manner consistent with the core concepts
>described in flow-based programming as originally documented by J.
>Paul Morrison in the 1970s.  This model lends itself well to visual
>diagramming, concurrency, componentization, testing, and reuse.  In
>addition to staying close to the fundamentals of flow-based
>programming, NiFi provides integration system specific features such
>as: guaranteed delivery; back pressure; ability to gracefully handle
>backlogs and data surges; and an operator interface that enables
>on-the-fly data flow generation, modification, and observation.
>
>== Rationale ==
>NiFi provides a reliable, scalable, manageable and accountable
>platform for developers and technical staff to create and evolve
>powerful data flows.  Such a system is useful in many contexts
>including large-scale enterprise integration, interaction with cloud
>services and frameworks, business to business, intra-departmental, and
>inter-departmental flows.  NiFi fits well within the Apache Software
>Foundation (ASF) family as it depends on numerous ASF projects and
>integrates with several others.  We also anticipate developing
>extensions for several other ASF projects such as Cassandra, Kafka,
>and Storm in the near future.
>
>== Initial Goals ==
>  * Ensure all dependencies are compliant with Apache License version
>2.0 and all that all code and documentation artifacts have the correct
>Apache licensing markings and notice.
>  * Establish a formal release process and schedule, allowing for
>dependable release cycles in a manner consistent with the Apache
>development process.
>  * Determine and establish a mechanism, possibly including a
>sub-project construct, that allows for extensions to the core
>application to occur at a pace that differs from the core application
>itself.
>
>== Current Status ==
>=== Meritocracy ===
>An integration platform is only as good as its ability to integrate
>systems in a reliable, timely, and repeatable manner.  The same can be
>said of its ability to attract talent and a variety of perspectives as
>integration systems by their nature are always evolving.  We will
>actively seek help and encourage promotion of influence in the project
>through meritocracy.
>
>=== Community ===
>Over the past several years, NiFi has developed a strong community of
>both developers and operators within the U.S. government.  We look
>forward to helping grow this to a broader base of industries.
>
>=== Core Developers ===
>The initial core developers are employed by the National Security
>Agency and defense contractors.  We will work to grow the community
>among a more diverse set of developers and industries.
>
>=== Alignment ===
>From its inception, NiFi was developed with an open source philosophy
>in mind and with the hopes of eventually being truly open sourced.
>The Apache way is consistent with the approach we have taken to date.
>The ASF clearly provides a mature and effective environment for
>successful development as is evident across the spectrum of well-known
>projects.  Further, NiFi depends on numerous ASF libraries and
>projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
>HttpClient, Jakarta and Maven.  We also anticipate extensions and
>dependencies with several more ASF projects, including Accumulo, Avro,
>Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
>This existing alignment with Apache and the desired community makes
>the Apache Incubator a good fit for NiFi.
>
>== Known Risks ==
>=== Orphaned Products ===
>Risk of orphaning is minimal.  The project user and developer base is
>substantial, growing, and there is already extensive operational use
>of NiFi.
>
>=== Inexperience with Open Source ===
>The initial committers to NiFi have limited experience with true open
>source software development.  However, despite the project origins
>being from closed source development we have modelled our behavior and
>community development on The Apache Way to the greatest extent
>possible.  This environment includes widely accessible source code
>repositories, published artifacts, ticket tracking, and extensive
>documentation. We also encourage contributions and frequent debate and
>hold regular, collaborative discussions through e-mail, chat rooms,
>and in-person meet-ups.  We are committed to the ideals of open source
>software and will eagerly seek out mentors and sponsors who can help
>us quickly come up to speed.
>
>=== Homogenous Developers ===
>The initial committers of NiFi come from a limited set of entities
>though we are committed to recruiting and developing additional
>committers from a broad spectrum of industries and backgrounds.
>
>=== Reliance on Salaried Developers ===
>We expect NiFi development to continue on salaried time and through
>volunteer time.  The initial committers are paid by their employers to
>contribute to this project.  We are committed to developing and
>recruiting participation from developers both salaried and
>non-salaried.
>
>=== Relationship with other Apache Projects ===
>As described in the alignment section, NiFi is already heavily
>dependent on other ASF projects and we anticipate further dependence
>and integration with new and emerging projects in the Apache family.
>
>=== An Excessive Fascination with the Apache Brand ===
>We respect the laudable Apache brand and that is certainly a factor in
>the decision to propose NiFi for the Apache Incubator.  We intend to
>provide a great deal of energy and capability to the ASF through this
>project.  We will be sensitive to and respectful of any overuse of the
>Apache brand and ensure our focus remains on how we benefit the Apache
>community.
>
>=== Documentation ===
>At this time there is no NiFi documentation on the web.  However, we
>have extensive documentation included within the application that
>details usage of the many functions.  We will be rapidly expanding the
>available documentation to cover things like installation, developer
>guide, frequently asked questions, best practices, and more.  This
>documentation will be posted to the NiFi wiki at apache.org.
>
>== Initial Source ==
>NiFi has been in active development since late 2006 with contributions
>from dozens of developers and feedback from hundreds of users and
>developers.  The core codebase is written in Java and includes
>detailed Javadocs and feature documentation.
>
>== Source and Intellectual Property Submission ==
>Previously referred to as Niagarafiles, the NiFi code and
>documentation materials will be submitted by the National Security
>Agency.  NiFi has been developed by a mix of government employees and
>private companies under government contract.  Material developed by
>the government employees is in the public domain and no U.S. copyright
>exists in works of the federal government.  For the contractor
>developed material in the initial submission, the U.S. Government has
>sufficient authority to open source per DFARS 252.227-7014.  NSA has
>submitted the Software Grant Agreement and Corporate Contributor
>License Agreement to the Apache Software Foundation.
>
>== External Dependencies ==
>We have at least one dependency on an LGPL library which we will
>promptly address.  Otherwise, we believe all current dependencies are
>compatible with the ASF guidelines.  Our dependency licenses come from
>the following license styles:  Apache v 2.0, BSD, Public Domain,
>Eclipse Public v1, MIT, CDDL v1.
>
>== Cryptography ==
>Consistent with http://www.apache.org/licenses/exports/ we believe
>NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
>cryptographic algorithms but is designed to use algorithms provided by
>Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
>These cryptographic algorithm providers are used to support SSL,
>SSH/SFTP, and the encryption and decryption of sensitive properties.
>In the event that it becomes necessary we will engage with appropriate
>Apache members to ensure we file any necessary paperwork or clarified
>any cryptographic export license concerns.
>
>== Required Resources ==
>=== Mailing Lists ===
>  * user@nifi.incubator.apache.org
>  * dev@nifi.incubator.apache.org
>  * private@nifi.incubator.apache.org
>  * commits@nifi.incubator.apache.org
>
>=== Source Control ===
>NiFi requests use of Git for source control
>(git://git.apache.org/nifi.git).  We request a writeable Git repo for
>NiFi with mirroring to be setup to Github through INFRA.  We request
>sponsor Benson Margulies (bimargulies) to assist with creating the
>INFRA ticket for this.
>
>=== Issue Tracking ===
>JIRA NiFi (NIFI)
>
>=== Initial Committers ===
>  * Brandon Devries <brandon.devries at gmail dot com>
>  * Matt Gilman <matt.c.gilman at gmail dot com>
>  * Tony Kurc <trkurc at gmail dot com>
>  * Mark Payne <markap14 at hotmail dot com>
>  * Adam Taft <adam at adamtaft dot com>
>  * Joseph Witt <joewitt at gmail dot com>
>
>=== Affiliations ===
>  * Brandon Devries (Requitest, Inc.)
>  * Matt Gilman (Raytheon)
>  * Tony Kurc (National Security Agency)
>  * Mark Payne (Sotera Defense Solutions, Inc.)
>  * Adam Taft (Requitest, Inc.)
>  * Joseph Witt (National Security Agency)
>
>== Sponsors ==
>=== Champion ===
>  * Benson Margulies (Basis Technology) <bimargulies at apache dot org>
>
>=== Nominated Mentors ===
>  * Drew Farris (Booz Allen Hamilton) <drew at apache dot org>
>  * Brock Noland (Cloudera) <brock at apache dot org>
>  * Billie Rinaldi (Hortonworks) <billie at apache dot org>
>
>=== Sponsoring Entity ===
>We request the Apache Incubator to sponsor this project.


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [PROPOSAL] NiFi for Incubation

[jan i <ja...@apache.org>]

On 20 November 2014 14:05, Joe Witt <jo...@gmail.com> wrote:

> Hadrian
>
> Yes there is a Software Grant Agreement.  NSA's tech transfer folks have
> already sent that to Apache.
>
> Given that we are coming from a closed source environment you won't find
> much.  That is what this proposal is about though as we're working hard to
> change that.
>
> The github link you reference has no relationship to this project.
>
> The relationship to Apache Camel will need to be explored further as NiFi
> is often used in similar problem spaces (integration).  Camel is really
> powerful in its core purpose and has an excellent community and a great
> deal of maturity.  NiFi provides a complete dataflow application with a
> major focus on the user experience, graphical creation and real-time
> command and control of those flows.  It will be interesting as we progress
> to see how we can best integrate with projects like Camel and I am looking
> forward to hearing some of the thoughts and ideas the community comes up.
>
Thanks for the explanation, but just to be sure, similar/overlapping
projects is not a problem per se, the only real concern is if 2 communities
can grow.

rgds
jan i.



>
> Thanks
> Joe
>
> On Thu, Nov 20, 2014 at 7:45 AM, Hadrian Zbarcea <hz...@gmail.com>
> wrote:
>
> > Sounds exciting. I have a couple of questions:
> >
> > 1. Is there a code grant? I assume so, the proposal states that the
> > project is active since 2006. What I could find [1] doesn't seem to be
> it.
> > 2. What is the overlap with Apache Camel (if any)?
> >
> > Cheers,
> > Hadrian
> >
> >
> > [1] https://github.com/Nifi
> >
> >
> >
> > On 11/19/2014 09:02 PM, Joe Witt wrote:
> >
> >> Hello,
> >>
> >>
> >> I would like to propose NiFi as an Apache Incubator Project.
> >>
> >> In addition to the copy provided below the Wiki version of the
> >> proposal can be found here:
> >> http://wiki.apache.org/incubator/NiFiProposal
> >>
> >> Thanks
> >>
> >> Joe
> >>
> >>
> >> = NiFi Proposal =
> >>
> >> == Abstract ==
> >> NiFi is a dataflow system based on the concepts of flow-based
> programming.
> >>
> >> == Proposal ==
> >> NiFi supports powerful and scalable directed graphs of data routing,
> >> transformation, and system mediation logic.  Some of the high-level
> >> capabilities and objectives of NiFi include:
> >>    * Web-based user interface for seamless experience between design,
> >> control, feedback, and monitoring of data flows
> >>    * Highly configurable along several dimensions of quality of service
> >> such as loss tolerant versus guaranteed delivery, low latency versus
> >> high throughput, and priority based queuing
> >>    * Fine-grained data provenance for all data received, forked,
> >> joined, cloned, modified, sent, and ultimately dropped as data reaches
> >> its configured end-state
> >>    * Component-based extension model along well defined interfaces
> >> enabling rapid development and effective testing
> >>
> >> == Background ==
> >> Reliable and effective dataflow between systems can be difficult
> >> whether you're running scripts on a laptop or have a massive
> >> distributed computing system operated by numerous teams and
> >> organizations.  As the volume and rate of data grows and as the number
> >> of systems, protocols, and formats increase and evolve so too does the
> >> complexity and need for greater insight and agility.  These are the
> >> dataflow challenges that NiFi was built to tackle.
> >>
> >> NiFi is designed in a manner consistent with the core concepts
> >> described in flow-based programming as originally documented by J.
> >> Paul Morrison in the 1970s.  This model lends itself well to visual
> >> diagramming, concurrency, componentization, testing, and reuse.  In
> >> addition to staying close to the fundamentals of flow-based
> >> programming, NiFi provides integration system specific features such
> >> as: guaranteed delivery; back pressure; ability to gracefully handle
> >> backlogs and data surges; and an operator interface that enables
> >> on-the-fly data flow generation, modification, and observation.
> >>
> >> == Rationale ==
> >> NiFi provides a reliable, scalable, manageable and accountable
> >> platform for developers and technical staff to create and evolve
> >> powerful data flows.  Such a system is useful in many contexts
> >> including large-scale enterprise integration, interaction with cloud
> >> services and frameworks, business to business, intra-departmental, and
> >> inter-departmental flows.  NiFi fits well within the Apache Software
> >> Foundation (ASF) family as it depends on numerous ASF projects and
> >> integrates with several others.  We also anticipate developing
> >> extensions for several other ASF projects such as Cassandra, Kafka,
> >> and Storm in the near future.
> >>
> >> == Initial Goals ==
> >>    * Ensure all dependencies are compliant with Apache License version
> >> 2.0 and all that all code and documentation artifacts have the correct
> >> Apache licensing markings and notice.
> >>    * Establish a formal release process and schedule, allowing for
> >> dependable release cycles in a manner consistent with the Apache
> >> development process.
> >>    * Determine and establish a mechanism, possibly including a
> >> sub-project construct, that allows for extensions to the core
> >> application to occur at a pace that differs from the core application
> >> itself.
> >>
> >> == Current Status ==
> >> === Meritocracy ===
> >> An integration platform is only as good as its ability to integrate
> >> systems in a reliable, timely, and repeatable manner.  The same can be
> >> said of its ability to attract talent and a variety of perspectives as
> >> integration systems by their nature are always evolving.  We will
> >> actively seek help and encourage promotion of influence in the project
> >> through meritocracy.
> >>
> >> === Community ===
> >> Over the past several years, NiFi has developed a strong community of
> >> both developers and operators within the U.S. government.  We look
> >> forward to helping grow this to a broader base of industries.
> >>
> >> === Core Developers ===
> >> The initial core developers are employed by the National Security
> >> Agency and defense contractors.  We will work to grow the community
> >> among a more diverse set of developers and industries.
> >>
> >> === Alignment ===
> >>  From its inception, NiFi was developed with an open source philosophy
> >> in mind and with the hopes of eventually being truly open sourced.
> >> The Apache way is consistent with the approach we have taken to date.
> >> The ASF clearly provides a mature and effective environment for
> >> successful development as is evident across the spectrum of well-known
> >> projects.  Further, NiFi depends on numerous ASF libraries and
> >> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
> >> HttpClient, Jakarta and Maven.  We also anticipate extensions and
> >> dependencies with several more ASF projects, including Accumulo, Avro,
> >> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
> >> This existing alignment with Apache and the desired community makes
> >> the Apache Incubator a good fit for NiFi.
> >>
> >> == Known Risks ==
> >> === Orphaned Products ===
> >> Risk of orphaning is minimal.  The project user and developer base is
> >> substantial, growing, and there is already extensive operational use
> >> of NiFi.
> >>
> >> === Inexperience with Open Source ===
> >> The initial committers to NiFi have limited experience with true open
> >> source software development.  However, despite the project origins
> >> being from closed source development we have modelled our behavior and
> >> community development on The Apache Way to the greatest extent
> >> possible.  This environment includes widely accessible source code
> >> repositories, published artifacts, ticket tracking, and extensive
> >> documentation. We also encourage contributions and frequent debate and
> >> hold regular, collaborative discussions through e-mail, chat rooms,
> >> and in-person meet-ups.  We are committed to the ideals of open source
> >> software and will eagerly seek out mentors and sponsors who can help
> >> us quickly come up to speed.
> >>
> >> === Homogenous Developers ===
> >> The initial committers of NiFi come from a limited set of entities
> >> though we are committed to recruiting and developing additional
> >> committers from a broad spectrum of industries and backgrounds.
> >>
> >> === Reliance on Salaried Developers ===
> >> We expect NiFi development to continue on salaried time and through
> >> volunteer time.  The initial committers are paid by their employers to
> >> contribute to this project.  We are committed to developing and
> >> recruiting participation from developers both salaried and
> >> non-salaried.
> >>
> >> === Relationship with other Apache Projects ===
> >> As described in the alignment section, NiFi is already heavily
> >> dependent on other ASF projects and we anticipate further dependence
> >> and integration with new and emerging projects in the Apache family.
> >>
> >> === An Excessive Fascination with the Apache Brand ===
> >> We respect the laudable Apache brand and that is certainly a factor in
> >> the decision to propose NiFi for the Apache Incubator.  We intend to
> >> provide a great deal of energy and capability to the ASF through this
> >> project.  We will be sensitive to and respectful of any overuse of the
> >> Apache brand and ensure our focus remains on how we benefit the Apache
> >> community.
> >>
> >> === Documentation ===
> >> At this time there is no NiFi documentation on the web.  However, we
> >> have extensive documentation included within the application that
> >> details usage of the many functions.  We will be rapidly expanding the
> >> available documentation to cover things like installation, developer
> >> guide, frequently asked questions, best practices, and more.  This
> >> documentation will be posted to the NiFi wiki at apache.org.
> >>
> >> == Initial Source ==
> >> NiFi has been in active development since late 2006 with contributions
> >> from dozens of developers and feedback from hundreds of users and
> >> developers.  The core codebase is written in Java and includes
> >> detailed Javadocs and feature documentation.
> >>
> >> == Source and Intellectual Property Submission ==
> >> Previously referred to as Niagarafiles, the NiFi code and
> >> documentation materials will be submitted by the National Security
> >> Agency.  NiFi has been developed by a mix of government employees and
> >> private companies under government contract.  Material developed by
> >> the government employees is in the public domain and no U.S. copyright
> >> exists in works of the federal government.  For the contractor
> >> developed material in the initial submission, the U.S. Government has
> >> sufficient authority to open source per DFARS 252.227-7014.  NSA has
> >> submitted the Software Grant Agreement and Corporate Contributor
> >> License Agreement to the Apache Software Foundation.
> >>
> >> == External Dependencies ==
> >> We have at least one dependency on an LGPL library which we will
> >> promptly address.  Otherwise, we believe all current dependencies are
> >> compatible with the ASF guidelines.  Our dependency licenses come from
> >> the following license styles:  Apache v 2.0, BSD, Public Domain,
> >> Eclipse Public v1, MIT, CDDL v1.
> >>
> >> == Cryptography ==
> >> Consistent with http://www.apache.org/licenses/exports/ we believe
> >> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
> >> cryptographic algorithms but is designed to use algorithms provided by
> >> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
> >> These cryptographic algorithm providers are used to support SSL,
> >> SSH/SFTP, and the encryption and decryption of sensitive properties.
> >> In the event that it becomes necessary we will engage with appropriate
> >> Apache members to ensure we file any necessary paperwork or clarified
> >> any cryptographic export license concerns.
> >>
> >> == Required Resources ==
> >> === Mailing Lists ===
> >>    * user@nifi.incubator.apache.org
> >>    * dev@nifi.incubator.apache.org
> >>    * private@nifi.incubator.apache.org
> >>    * commits@nifi.incubator.apache.org
> >>
> >> === Source Control ===
> >> NiFi requests use of Git for source control
> >> (git://git.apache.org/nifi.git).  We request a writeable Git repo for
> >> NiFi with mirroring to be setup to Github through INFRA.  We request
> >> sponsor Benson Margulies (bimargulies) to assist with creating the
> >> INFRA ticket for this.
> >>
> >> === Issue Tracking ===
> >> JIRA NiFi (NIFI)
> >>
> >> === Initial Committers ===
> >>    * Brandon Devries <brandon.devries at gmail dot com>
> >>    * Matt Gilman <matt.c.gilman at gmail dot com>
> >>    * Tony Kurc <trkurc at gmail dot com>
> >>    * Mark Payne <markap14 at hotmail dot com>
> >>    * Adam Taft <adam at adamtaft dot com>
> >>    * Joseph Witt <joewitt at gmail dot com>
> >>
> >> === Affiliations ===
> >>    * Brandon Devries (Requitest, Inc.)
> >>    * Matt Gilman (Raytheon)
> >>    * Tony Kurc (National Security Agency)
> >>    * Mark Payne (Sotera Defense Solutions, Inc.)
> >>    * Adam Taft (Requitest, Inc.)
> >>    * Joseph Witt (National Security Agency)
> >>
> >> == Sponsors ==
> >> === Champion ===
> >>    * Benson Margulies (Basis Technology) <bimargulies at apache dot org>
> >>
> >> === Nominated Mentors ===
> >>    * Drew Farris (Booz Allen Hamilton) <drew at apache dot org>
> >>    * Brock Noland (Cloudera) <brock at apache dot org>
> >>    * Billie Rinaldi (Hortonworks) <billie at apache dot org>
> >>
> >> === Sponsoring Entity ===
> >> We request the Apache Incubator to sponsor this project.
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>

Re: [PROPOSAL] NiFi for Incubation

[Joe Witt <jo...@gmail.com>]

Hadrian

Yes there is a Software Grant Agreement.  NSA's tech transfer folks have
already sent that to Apache.

Given that we are coming from a closed source environment you won't find
much.  That is what this proposal is about though as we're working hard to
change that.

The github link you reference has no relationship to this project.

The relationship to Apache Camel will need to be explored further as NiFi
is often used in similar problem spaces (integration).  Camel is really
powerful in its core purpose and has an excellent community and a great
deal of maturity.  NiFi provides a complete dataflow application with a
major focus on the user experience, graphical creation and real-time
command and control of those flows.  It will be interesting as we progress
to see how we can best integrate with projects like Camel and I am looking
forward to hearing some of the thoughts and ideas the community comes up.

Thanks
Joe

On Thu, Nov 20, 2014 at 7:45 AM, Hadrian Zbarcea <hz...@gmail.com> wrote:

> Sounds exciting. I have a couple of questions:
>
> 1. Is there a code grant? I assume so, the proposal states that the
> project is active since 2006. What I could find [1] doesn't seem to be it.
> 2. What is the overlap with Apache Camel (if any)?
>
> Cheers,
> Hadrian
>
>
> [1] https://github.com/Nifi
>
>
>
> On 11/19/2014 09:02 PM, Joe Witt wrote:
>
>> Hello,
>>
>>
>> I would like to propose NiFi as an Apache Incubator Project.
>>
>> In addition to the copy provided below the Wiki version of the
>> proposal can be found here:
>> http://wiki.apache.org/incubator/NiFiProposal
>>
>> Thanks
>>
>> Joe
>>
>>
>> = NiFi Proposal =
>>
>> == Abstract ==
>> NiFi is a dataflow system based on the concepts of flow-based programming.
>>
>> == Proposal ==
>> NiFi supports powerful and scalable directed graphs of data routing,
>> transformation, and system mediation logic.  Some of the high-level
>> capabilities and objectives of NiFi include:
>>    * Web-based user interface for seamless experience between design,
>> control, feedback, and monitoring of data flows
>>    * Highly configurable along several dimensions of quality of service
>> such as loss tolerant versus guaranteed delivery, low latency versus
>> high throughput, and priority based queuing
>>    * Fine-grained data provenance for all data received, forked,
>> joined, cloned, modified, sent, and ultimately dropped as data reaches
>> its configured end-state
>>    * Component-based extension model along well defined interfaces
>> enabling rapid development and effective testing
>>
>> == Background ==
>> Reliable and effective dataflow between systems can be difficult
>> whether you're running scripts on a laptop or have a massive
>> distributed computing system operated by numerous teams and
>> organizations.  As the volume and rate of data grows and as the number
>> of systems, protocols, and formats increase and evolve so too does the
>> complexity and need for greater insight and agility.  These are the
>> dataflow challenges that NiFi was built to tackle.
>>
>> NiFi is designed in a manner consistent with the core concepts
>> described in flow-based programming as originally documented by J.
>> Paul Morrison in the 1970s.  This model lends itself well to visual
>> diagramming, concurrency, componentization, testing, and reuse.  In
>> addition to staying close to the fundamentals of flow-based
>> programming, NiFi provides integration system specific features such
>> as: guaranteed delivery; back pressure; ability to gracefully handle
>> backlogs and data surges; and an operator interface that enables
>> on-the-fly data flow generation, modification, and observation.
>>
>> == Rationale ==
>> NiFi provides a reliable, scalable, manageable and accountable
>> platform for developers and technical staff to create and evolve
>> powerful data flows.  Such a system is useful in many contexts
>> including large-scale enterprise integration, interaction with cloud
>> services and frameworks, business to business, intra-departmental, and
>> inter-departmental flows.  NiFi fits well within the Apache Software
>> Foundation (ASF) family as it depends on numerous ASF projects and
>> integrates with several others.  We also anticipate developing
>> extensions for several other ASF projects such as Cassandra, Kafka,
>> and Storm in the near future.
>>
>> == Initial Goals ==
>>    * Ensure all dependencies are compliant with Apache License version
>> 2.0 and all that all code and documentation artifacts have the correct
>> Apache licensing markings and notice.
>>    * Establish a formal release process and schedule, allowing for
>> dependable release cycles in a manner consistent with the Apache
>> development process.
>>    * Determine and establish a mechanism, possibly including a
>> sub-project construct, that allows for extensions to the core
>> application to occur at a pace that differs from the core application
>> itself.
>>
>> == Current Status ==
>> === Meritocracy ===
>> An integration platform is only as good as its ability to integrate
>> systems in a reliable, timely, and repeatable manner.  The same can be
>> said of its ability to attract talent and a variety of perspectives as
>> integration systems by their nature are always evolving.  We will
>> actively seek help and encourage promotion of influence in the project
>> through meritocracy.
>>
>> === Community ===
>> Over the past several years, NiFi has developed a strong community of
>> both developers and operators within the U.S. government.  We look
>> forward to helping grow this to a broader base of industries.
>>
>> === Core Developers ===
>> The initial core developers are employed by the National Security
>> Agency and defense contractors.  We will work to grow the community
>> among a more diverse set of developers and industries.
>>
>> === Alignment ===
>>  From its inception, NiFi was developed with an open source philosophy
>> in mind and with the hopes of eventually being truly open sourced.
>> The Apache way is consistent with the approach we have taken to date.
>> The ASF clearly provides a mature and effective environment for
>> successful development as is evident across the spectrum of well-known
>> projects.  Further, NiFi depends on numerous ASF libraries and
>> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
>> HttpClient, Jakarta and Maven.  We also anticipate extensions and
>> dependencies with several more ASF projects, including Accumulo, Avro,
>> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
>> This existing alignment with Apache and the desired community makes
>> the Apache Incubator a good fit for NiFi.
>>
>> == Known Risks ==
>> === Orphaned Products ===
>> Risk of orphaning is minimal.  The project user and developer base is
>> substantial, growing, and there is already extensive operational use
>> of NiFi.
>>
>> === Inexperience with Open Source ===
>> The initial committers to NiFi have limited experience with true open
>> source software development.  However, despite the project origins
>> being from closed source development we have modelled our behavior and
>> community development on The Apache Way to the greatest extent
>> possible.  This environment includes widely accessible source code
>> repositories, published artifacts, ticket tracking, and extensive
>> documentation. We also encourage contributions and frequent debate and
>> hold regular, collaborative discussions through e-mail, chat rooms,
>> and in-person meet-ups.  We are committed to the ideals of open source
>> software and will eagerly seek out mentors and sponsors who can help
>> us quickly come up to speed.
>>
>> === Homogenous Developers ===
>> The initial committers of NiFi come from a limited set of entities
>> though we are committed to recruiting and developing additional
>> committers from a broad spectrum of industries and backgrounds.
>>
>> === Reliance on Salaried Developers ===
>> We expect NiFi development to continue on salaried time and through
>> volunteer time.  The initial committers are paid by their employers to
>> contribute to this project.  We are committed to developing and
>> recruiting participation from developers both salaried and
>> non-salaried.
>>
>> === Relationship with other Apache Projects ===
>> As described in the alignment section, NiFi is already heavily
>> dependent on other ASF projects and we anticipate further dependence
>> and integration with new and emerging projects in the Apache family.
>>
>> === An Excessive Fascination with the Apache Brand ===
>> We respect the laudable Apache brand and that is certainly a factor in
>> the decision to propose NiFi for the Apache Incubator.  We intend to
>> provide a great deal of energy and capability to the ASF through this
>> project.  We will be sensitive to and respectful of any overuse of the
>> Apache brand and ensure our focus remains on how we benefit the Apache
>> community.
>>
>> === Documentation ===
>> At this time there is no NiFi documentation on the web.  However, we
>> have extensive documentation included within the application that
>> details usage of the many functions.  We will be rapidly expanding the
>> available documentation to cover things like installation, developer
>> guide, frequently asked questions, best practices, and more.  This
>> documentation will be posted to the NiFi wiki at apache.org.
>>
>> == Initial Source ==
>> NiFi has been in active development since late 2006 with contributions
>> from dozens of developers and feedback from hundreds of users and
>> developers.  The core codebase is written in Java and includes
>> detailed Javadocs and feature documentation.
>>
>> == Source and Intellectual Property Submission ==
>> Previously referred to as Niagarafiles, the NiFi code and
>> documentation materials will be submitted by the National Security
>> Agency.  NiFi has been developed by a mix of government employees and
>> private companies under government contract.  Material developed by
>> the government employees is in the public domain and no U.S. copyright
>> exists in works of the federal government.  For the contractor
>> developed material in the initial submission, the U.S. Government has
>> sufficient authority to open source per DFARS 252.227-7014.  NSA has
>> submitted the Software Grant Agreement and Corporate Contributor
>> License Agreement to the Apache Software Foundation.
>>
>> == External Dependencies ==
>> We have at least one dependency on an LGPL library which we will
>> promptly address.  Otherwise, we believe all current dependencies are
>> compatible with the ASF guidelines.  Our dependency licenses come from
>> the following license styles:  Apache v 2.0, BSD, Public Domain,
>> Eclipse Public v1, MIT, CDDL v1.
>>
>> == Cryptography ==
>> Consistent with http://www.apache.org/licenses/exports/ we believe
>> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
>> cryptographic algorithms but is designed to use algorithms provided by
>> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
>> These cryptographic algorithm providers are used to support SSL,
>> SSH/SFTP, and the encryption and decryption of sensitive properties.
>> In the event that it becomes necessary we will engage with appropriate
>> Apache members to ensure we file any necessary paperwork or clarified
>> any cryptographic export license concerns.
>>
>> == Required Resources ==
>> === Mailing Lists ===
>>    * user@nifi.incubator.apache.org
>>    * dev@nifi.incubator.apache.org
>>    * private@nifi.incubator.apache.org
>>    * commits@nifi.incubator.apache.org
>>
>> === Source Control ===
>> NiFi requests use of Git for source control
>> (git://git.apache.org/nifi.git).  We request a writeable Git repo for
>> NiFi with mirroring to be setup to Github through INFRA.  We request
>> sponsor Benson Margulies (bimargulies) to assist with creating the
>> INFRA ticket for this.
>>
>> === Issue Tracking ===
>> JIRA NiFi (NIFI)
>>
>> === Initial Committers ===
>>    * Brandon Devries <brandon.devries at gmail dot com>
>>    * Matt Gilman <matt.c.gilman at gmail dot com>
>>    * Tony Kurc <trkurc at gmail dot com>
>>    * Mark Payne <markap14 at hotmail dot com>
>>    * Adam Taft <adam at adamtaft dot com>
>>    * Joseph Witt <joewitt at gmail dot com>
>>
>> === Affiliations ===
>>    * Brandon Devries (Requitest, Inc.)
>>    * Matt Gilman (Raytheon)
>>    * Tony Kurc (National Security Agency)
>>    * Mark Payne (Sotera Defense Solutions, Inc.)
>>    * Adam Taft (Requitest, Inc.)
>>    * Joseph Witt (National Security Agency)
>>
>> == Sponsors ==
>> === Champion ===
>>    * Benson Margulies (Basis Technology) <bimargulies at apache dot org>
>>
>> === Nominated Mentors ===
>>    * Drew Farris (Booz Allen Hamilton) <drew at apache dot org>
>>    * Brock Noland (Cloudera) <brock at apache dot org>
>>    * Billie Rinaldi (Hortonworks) <billie at apache dot org>
>>
>> === Sponsoring Entity ===
>> We request the Apache Incubator to sponsor this project.
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [PROPOSAL] NiFi for Incubation

[Hadrian Zbarcea <hz...@gmail.com>]

Sounds exciting. I have a couple of questions:

1. Is there a code grant? I assume so, the proposal states that the 
project is active since 2006. What I could find [1] doesn't seem to be it.
2. What is the overlap with Apache Camel (if any)?

Cheers,
Hadrian


[1] https://github.com/Nifi


On 11/19/2014 09:02 PM, Joe Witt wrote:
> Hello,
>
>
> I would like to propose NiFi as an Apache Incubator Project.
>
> In addition to the copy provided below the Wiki version of the
> proposal can be found here:
> http://wiki.apache.org/incubator/NiFiProposal
>
> Thanks
>
> Joe
>
>
> = NiFi Proposal =
>
> == Abstract ==
> NiFi is a dataflow system based on the concepts of flow-based programming.
>
> == Proposal ==
> NiFi supports powerful and scalable directed graphs of data routing,
> transformation, and system mediation logic.  Some of the high-level
> capabilities and objectives of NiFi include:
>    * Web-based user interface for seamless experience between design,
> control, feedback, and monitoring of data flows
>    * Highly configurable along several dimensions of quality of service
> such as loss tolerant versus guaranteed delivery, low latency versus
> high throughput, and priority based queuing
>    * Fine-grained data provenance for all data received, forked,
> joined, cloned, modified, sent, and ultimately dropped as data reaches
> its configured end-state
>    * Component-based extension model along well defined interfaces
> enabling rapid development and effective testing
>
> == Background ==
> Reliable and effective dataflow between systems can be difficult
> whether you're running scripts on a laptop or have a massive
> distributed computing system operated by numerous teams and
> organizations.  As the volume and rate of data grows and as the number
> of systems, protocols, and formats increase and evolve so too does the
> complexity and need for greater insight and agility.  These are the
> dataflow challenges that NiFi was built to tackle.
>
> NiFi is designed in a manner consistent with the core concepts
> described in flow-based programming as originally documented by J.
> Paul Morrison in the 1970s.  This model lends itself well to visual
> diagramming, concurrency, componentization, testing, and reuse.  In
> addition to staying close to the fundamentals of flow-based
> programming, NiFi provides integration system specific features such
> as: guaranteed delivery; back pressure; ability to gracefully handle
> backlogs and data surges; and an operator interface that enables
> on-the-fly data flow generation, modification, and observation.
>
> == Rationale ==
> NiFi provides a reliable, scalable, manageable and accountable
> platform for developers and technical staff to create and evolve
> powerful data flows.  Such a system is useful in many contexts
> including large-scale enterprise integration, interaction with cloud
> services and frameworks, business to business, intra-departmental, and
> inter-departmental flows.  NiFi fits well within the Apache Software
> Foundation (ASF) family as it depends on numerous ASF projects and
> integrates with several others.  We also anticipate developing
> extensions for several other ASF projects such as Cassandra, Kafka,
> and Storm in the near future.
>
> == Initial Goals ==
>    * Ensure all dependencies are compliant with Apache License version
> 2.0 and all that all code and documentation artifacts have the correct
> Apache licensing markings and notice.
>    * Establish a formal release process and schedule, allowing for
> dependable release cycles in a manner consistent with the Apache
> development process.
>    * Determine and establish a mechanism, possibly including a
> sub-project construct, that allows for extensions to the core
> application to occur at a pace that differs from the core application
> itself.
>
> == Current Status ==
> === Meritocracy ===
> An integration platform is only as good as its ability to integrate
> systems in a reliable, timely, and repeatable manner.  The same can be
> said of its ability to attract talent and a variety of perspectives as
> integration systems by their nature are always evolving.  We will
> actively seek help and encourage promotion of influence in the project
> through meritocracy.
>
> === Community ===
> Over the past several years, NiFi has developed a strong community of
> both developers and operators within the U.S. government.  We look
> forward to helping grow this to a broader base of industries.
>
> === Core Developers ===
> The initial core developers are employed by the National Security
> Agency and defense contractors.  We will work to grow the community
> among a more diverse set of developers and industries.
>
> === Alignment ===
>  From its inception, NiFi was developed with an open source philosophy
> in mind and with the hopes of eventually being truly open sourced.
> The Apache way is consistent with the approach we have taken to date.
> The ASF clearly provides a mature and effective environment for
> successful development as is evident across the spectrum of well-known
> projects.  Further, NiFi depends on numerous ASF libraries and
> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
> HttpClient, Jakarta and Maven.  We also anticipate extensions and
> dependencies with several more ASF projects, including Accumulo, Avro,
> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
> This existing alignment with Apache and the desired community makes
> the Apache Incubator a good fit for NiFi.
>
> == Known Risks ==
> === Orphaned Products ===
> Risk of orphaning is minimal.  The project user and developer base is
> substantial, growing, and there is already extensive operational use
> of NiFi.
>
> === Inexperience with Open Source ===
> The initial committers to NiFi have limited experience with true open
> source software development.  However, despite the project origins
> being from closed source development we have modelled our behavior and
> community development on The Apache Way to the greatest extent
> possible.  This environment includes widely accessible source code
> repositories, published artifacts, ticket tracking, and extensive
> documentation. We also encourage contributions and frequent debate and
> hold regular, collaborative discussions through e-mail, chat rooms,
> and in-person meet-ups.  We are committed to the ideals of open source
> software and will eagerly seek out mentors and sponsors who can help
> us quickly come up to speed.
>
> === Homogenous Developers ===
> The initial committers of NiFi come from a limited set of entities
> though we are committed to recruiting and developing additional
> committers from a broad spectrum of industries and backgrounds.
>
> === Reliance on Salaried Developers ===
> We expect NiFi development to continue on salaried time and through
> volunteer time.  The initial committers are paid by their employers to
> contribute to this project.  We are committed to developing and
> recruiting participation from developers both salaried and
> non-salaried.
>
> === Relationship with other Apache Projects ===
> As described in the alignment section, NiFi is already heavily
> dependent on other ASF projects and we anticipate further dependence
> and integration with new and emerging projects in the Apache family.
>
> === An Excessive Fascination with the Apache Brand ===
> We respect the laudable Apache brand and that is certainly a factor in
> the decision to propose NiFi for the Apache Incubator.  We intend to
> provide a great deal of energy and capability to the ASF through this
> project.  We will be sensitive to and respectful of any overuse of the
> Apache brand and ensure our focus remains on how we benefit the Apache
> community.
>
> === Documentation ===
> At this time there is no NiFi documentation on the web.  However, we
> have extensive documentation included within the application that
> details usage of the many functions.  We will be rapidly expanding the
> available documentation to cover things like installation, developer
> guide, frequently asked questions, best practices, and more.  This
> documentation will be posted to the NiFi wiki at apache.org.
>
> == Initial Source ==
> NiFi has been in active development since late 2006 with contributions
> from dozens of developers and feedback from hundreds of users and
> developers.  The core codebase is written in Java and includes
> detailed Javadocs and feature documentation.
>
> == Source and Intellectual Property Submission ==
> Previously referred to as Niagarafiles, the NiFi code and
> documentation materials will be submitted by the National Security
> Agency.  NiFi has been developed by a mix of government employees and
> private companies under government contract.  Material developed by
> the government employees is in the public domain and no U.S. copyright
> exists in works of the federal government.  For the contractor
> developed material in the initial submission, the U.S. Government has
> sufficient authority to open source per DFARS 252.227-7014.  NSA has
> submitted the Software Grant Agreement and Corporate Contributor
> License Agreement to the Apache Software Foundation.
>
> == External Dependencies ==
> We have at least one dependency on an LGPL library which we will
> promptly address.  Otherwise, we believe all current dependencies are
> compatible with the ASF guidelines.  Our dependency licenses come from
> the following license styles:  Apache v 2.0, BSD, Public Domain,
> Eclipse Public v1, MIT, CDDL v1.
>
> == Cryptography ==
> Consistent with http://www.apache.org/licenses/exports/ we believe
> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
> cryptographic algorithms but is designed to use algorithms provided by
> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
> These cryptographic algorithm providers are used to support SSL,
> SSH/SFTP, and the encryption and decryption of sensitive properties.
> In the event that it becomes necessary we will engage with appropriate
> Apache members to ensure we file any necessary paperwork or clarified
> any cryptographic export license concerns.
>
> == Required Resources ==
> === Mailing Lists ===
>    * user@nifi.incubator.apache.org
>    * dev@nifi.incubator.apache.org
>    * private@nifi.incubator.apache.org
>    * commits@nifi.incubator.apache.org
>
> === Source Control ===
> NiFi requests use of Git for source control
> (git://git.apache.org/nifi.git).  We request a writeable Git repo for
> NiFi with mirroring to be setup to Github through INFRA.  We request
> sponsor Benson Margulies (bimargulies) to assist with creating the
> INFRA ticket for this.
>
> === Issue Tracking ===
> JIRA NiFi (NIFI)
>
> === Initial Committers ===
>    * Brandon Devries <brandon.devries at gmail dot com>
>    * Matt Gilman <matt.c.gilman at gmail dot com>
>    * Tony Kurc <trkurc at gmail dot com>
>    * Mark Payne <markap14 at hotmail dot com>
>    * Adam Taft <adam at adamtaft dot com>
>    * Joseph Witt <joewitt at gmail dot com>
>
> === Affiliations ===
>    * Brandon Devries (Requitest, Inc.)
>    * Matt Gilman (Raytheon)
>    * Tony Kurc (National Security Agency)
>    * Mark Payne (Sotera Defense Solutions, Inc.)
>    * Adam Taft (Requitest, Inc.)
>    * Joseph Witt (National Security Agency)
>
> == Sponsors ==
> === Champion ===
>    * Benson Margulies (Basis Technology) <bimargulies at apache dot org>
>
> === Nominated Mentors ===
>    * Drew Farris (Booz Allen Hamilton) <drew at apache dot org>
>    * Brock Noland (Cloudera) <brock at apache dot org>
>    * Billie Rinaldi (Hortonworks) <billie at apache dot org>
>
> === Sponsoring Entity ===
> We request the Apache Incubator to sponsor this project.
>


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [PROPOSAL] NiFi for Incubation

[Benson Margulies <bi...@gmail.com>]

I've advised Joe to 'asterix' the would-be mentors who are not iPMC yet, so
that he can proceed to a vote on the base of the ones who are sooner rather
than later, and the stragglers can be formally added to the metadata once
they are on the iPMC.

Re: [PROPOSAL] NiFi for Incubation

[Bertrand Delacretaz <bd...@apache.org>]

Hi,

On Fri, Nov 21, 2014 at 2:23 PM, Joe Witt <jo...@gmail.com> wrote:
> ...I had noticed other proposals listing these folks as mentors
> but also providing their IPMC status along side.  I'll do the same for now
> and if they join the ipmc i'll update it and if they end up not doing so
> will remove them.  If that is not reasonable then just let me know....

That's perfectly fine, thanks!

-Bertrand

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [PROPOSAL] NiFi for Incubation

[Joe Witt <jo...@gmail.com>]

Bertrand,

Understood.  I had noticed other proposals listing these folks as mentors
but also providing their IPMC status along side.  I'll do the same for now
and if they join the ipmc i'll update it and if they end up not doing so
will remove them.  If that is not reasonable then just let me know.

Thanks
Joe

On Fri, Nov 21, 2014 at 7:59 AM, Bertrand Delacretaz <bdelacretaz@apache.org
> wrote:

> Hi,
>
> On Fri, Nov 21, 2014 at 1:52 PM, Joe Witt <jo...@gmail.com> wrote:
> ...
> > http://wiki.apache.org/incubator/NiFiProposal
>
> Note that AFAICS some of the nominated mentors are not members of the
> Incubator PMC [1] - they cannot be "formal" incubation mentors then.
>
> ASF members can join that PMC just by asking.
>
> ASF committers can be elected to that PMC with the support of ASF or
> Incubator PMC members, subject to an Incubator PMC vote, that can be
> discussed on the incubator private list.
>
> Others can help with mentoring activities of course - and that's a
> great way of eventually joining this PMC if you're not an ASF member.
> But they won't be listed as mentors on the proposal or podling status
> page.
>
> -Bertrand
>
> [1] http://people.apache.org/committers-by-project.html#incubator-pmc
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [PROPOSAL] NiFi for Incubation

[Bertrand Delacretaz <bd...@apache.org>]

Hi,

On Fri, Nov 21, 2014 at 1:52 PM, Joe Witt <jo...@gmail.com> wrote:
...
> http://wiki.apache.org/incubator/NiFiProposal

Note that AFAICS some of the nominated mentors are not members of the
Incubator PMC [1] - they cannot be "formal" incubation mentors then.

ASF members can join that PMC just by asking.

ASF committers can be elected to that PMC with the support of ASF or
Incubator PMC members, subject to an Incubator PMC vote, that can be
discussed on the incubator private list.

Others can help with mentoring activities of course - and that's a
great way of eventually joining this PMC if you're not an ASF member.
But they won't be listed as mentors on the proposal or podling status
page.

-Bertrand

[1] http://people.apache.org/committers-by-project.html#incubator-pmc

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [PROPOSAL] NiFi for Incubation

[Joe Witt <jo...@gmail.com>]

Sergio

Excellent - really appreciate it and look forward to working with you.
Have updated the proposal.
  Sergio Fernandez (Redlink, Salzburg Research> <wikier at apache dot org>

http://wiki.apache.org/incubator/NiFiProposal

Thanks
Joe

On Fri, Nov 21, 2014 at 7:30 AM, Sergio Fernández <
sergio.fernandez@salzburgresearch.at> wrote:

> Very interesting project!
> I'm happy to help mentoring it.
>
>
> On 20/11/14 03:02, Joe Witt wrote:
>
>> Hello,
>>
>>
>> I would like to propose NiFi as an Apache Incubator Project.
>>
>> In addition to the copy provided below the Wiki version of the
>> proposal can be found here:
>> http://wiki.apache.org/incubator/NiFiProposal
>>
>> Thanks
>>
>> Joe
>>
>>
>> = NiFi Proposal =
>>
>> == Abstract ==
>> NiFi is a dataflow system based on the concepts of flow-based programming.
>>
>> == Proposal ==
>> NiFi supports powerful and scalable directed graphs of data routing,
>> transformation, and system mediation logic.  Some of the high-level
>> capabilities and objectives of NiFi include:
>>    * Web-based user interface for seamless experience between design,
>> control, feedback, and monitoring of data flows
>>    * Highly configurable along several dimensions of quality of service
>> such as loss tolerant versus guaranteed delivery, low latency versus
>> high throughput, and priority based queuing
>>    * Fine-grained data provenance for all data received, forked,
>> joined, cloned, modified, sent, and ultimately dropped as data reaches
>> its configured end-state
>>    * Component-based extension model along well defined interfaces
>> enabling rapid development and effective testing
>>
>> == Background ==
>> Reliable and effective dataflow between systems can be difficult
>> whether you're running scripts on a laptop or have a massive
>> distributed computing system operated by numerous teams and
>> organizations.  As the volume and rate of data grows and as the number
>> of systems, protocols, and formats increase and evolve so too does the
>> complexity and need for greater insight and agility.  These are the
>> dataflow challenges that NiFi was built to tackle.
>>
>> NiFi is designed in a manner consistent with the core concepts
>> described in flow-based programming as originally documented by J.
>> Paul Morrison in the 1970s.  This model lends itself well to visual
>> diagramming, concurrency, componentization, testing, and reuse.  In
>> addition to staying close to the fundamentals of flow-based
>> programming, NiFi provides integration system specific features such
>> as: guaranteed delivery; back pressure; ability to gracefully handle
>> backlogs and data surges; and an operator interface that enables
>> on-the-fly data flow generation, modification, and observation.
>>
>> == Rationale ==
>> NiFi provides a reliable, scalable, manageable and accountable
>> platform for developers and technical staff to create and evolve
>> powerful data flows.  Such a system is useful in many contexts
>> including large-scale enterprise integration, interaction with cloud
>> services and frameworks, business to business, intra-departmental, and
>> inter-departmental flows.  NiFi fits well within the Apache Software
>> Foundation (ASF) family as it depends on numerous ASF projects and
>> integrates with several others.  We also anticipate developing
>> extensions for several other ASF projects such as Cassandra, Kafka,
>> and Storm in the near future.
>>
>> == Initial Goals ==
>>    * Ensure all dependencies are compliant with Apache License version
>> 2.0 and all that all code and documentation artifacts have the correct
>> Apache licensing markings and notice.
>>    * Establish a formal release process and schedule, allowing for
>> dependable release cycles in a manner consistent with the Apache
>> development process.
>>    * Determine and establish a mechanism, possibly including a
>> sub-project construct, that allows for extensions to the core
>> application to occur at a pace that differs from the core application
>> itself.
>>
>> == Current Status ==
>> === Meritocracy ===
>> An integration platform is only as good as its ability to integrate
>> systems in a reliable, timely, and repeatable manner.  The same can be
>> said of its ability to attract talent and a variety of perspectives as
>> integration systems by their nature are always evolving.  We will
>> actively seek help and encourage promotion of influence in the project
>> through meritocracy.
>>
>> === Community ===
>> Over the past several years, NiFi has developed a strong community of
>> both developers and operators within the U.S. government.  We look
>> forward to helping grow this to a broader base of industries.
>>
>> === Core Developers ===
>> The initial core developers are employed by the National Security
>> Agency and defense contractors.  We will work to grow the community
>> among a more diverse set of developers and industries.
>>
>> === Alignment ===
>>  From its inception, NiFi was developed with an open source philosophy
>> in mind and with the hopes of eventually being truly open sourced.
>> The Apache way is consistent with the approach we have taken to date.
>> The ASF clearly provides a mature and effective environment for
>> successful development as is evident across the spectrum of well-known
>> projects.  Further, NiFi depends on numerous ASF libraries and
>> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
>> HttpClient, Jakarta and Maven.  We also anticipate extensions and
>> dependencies with several more ASF projects, including Accumulo, Avro,
>> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
>> This existing alignment with Apache and the desired community makes
>> the Apache Incubator a good fit for NiFi.
>>
>> == Known Risks ==
>> === Orphaned Products ===
>> Risk of orphaning is minimal.  The project user and developer base is
>> substantial, growing, and there is already extensive operational use
>> of NiFi.
>>
>> === Inexperience with Open Source ===
>> The initial committers to NiFi have limited experience with true open
>> source software development.  However, despite the project origins
>> being from closed source development we have modelled our behavior and
>> community development on The Apache Way to the greatest extent
>> possible.  This environment includes widely accessible source code
>> repositories, published artifacts, ticket tracking, and extensive
>> documentation. We also encourage contributions and frequent debate and
>> hold regular, collaborative discussions through e-mail, chat rooms,
>> and in-person meet-ups.  We are committed to the ideals of open source
>> software and will eagerly seek out mentors and sponsors who can help
>> us quickly come up to speed.
>>
>> === Homogenous Developers ===
>> The initial committers of NiFi come from a limited set of entities
>> though we are committed to recruiting and developing additional
>> committers from a broad spectrum of industries and backgrounds.
>>
>> === Reliance on Salaried Developers ===
>> We expect NiFi development to continue on salaried time and through
>> volunteer time.  The initial committers are paid by their employers to
>> contribute to this project.  We are committed to developing and
>> recruiting participation from developers both salaried and
>> non-salaried.
>>
>> === Relationship with other Apache Projects ===
>> As described in the alignment section, NiFi is already heavily
>> dependent on other ASF projects and we anticipate further dependence
>> and integration with new and emerging projects in the Apache family.
>>
>> === An Excessive Fascination with the Apache Brand ===
>> We respect the laudable Apache brand and that is certainly a factor in
>> the decision to propose NiFi for the Apache Incubator.  We intend to
>> provide a great deal of energy and capability to the ASF through this
>> project.  We will be sensitive to and respectful of any overuse of the
>> Apache brand and ensure our focus remains on how we benefit the Apache
>> community.
>>
>> === Documentation ===
>> At this time there is no NiFi documentation on the web.  However, we
>> have extensive documentation included within the application that
>> details usage of the many functions.  We will be rapidly expanding the
>> available documentation to cover things like installation, developer
>> guide, frequently asked questions, best practices, and more.  This
>> documentation will be posted to the NiFi wiki at apache.org.
>>
>> == Initial Source ==
>> NiFi has been in active development since late 2006 with contributions
>> from dozens of developers and feedback from hundreds of users and
>> developers.  The core codebase is written in Java and includes
>> detailed Javadocs and feature documentation.
>>
>> == Source and Intellectual Property Submission ==
>> Previously referred to as Niagarafiles, the NiFi code and
>> documentation materials will be submitted by the National Security
>> Agency.  NiFi has been developed by a mix of government employees and
>> private companies under government contract.  Material developed by
>> the government employees is in the public domain and no U.S. copyright
>> exists in works of the federal government.  For the contractor
>> developed material in the initial submission, the U.S. Government has
>> sufficient authority to open source per DFARS 252.227-7014.  NSA has
>> submitted the Software Grant Agreement and Corporate Contributor
>> License Agreement to the Apache Software Foundation.
>>
>> == External Dependencies ==
>> We have at least one dependency on an LGPL library which we will
>> promptly address.  Otherwise, we believe all current dependencies are
>> compatible with the ASF guidelines.  Our dependency licenses come from
>> the following license styles:  Apache v 2.0, BSD, Public Domain,
>> Eclipse Public v1, MIT, CDDL v1.
>>
>> == Cryptography ==
>> Consistent with http://www.apache.org/licenses/exports/ we believe
>> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
>> cryptographic algorithms but is designed to use algorithms provided by
>> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
>> These cryptographic algorithm providers are used to support SSL,
>> SSH/SFTP, and the encryption and decryption of sensitive properties.
>> In the event that it becomes necessary we will engage with appropriate
>> Apache members to ensure we file any necessary paperwork or clarified
>> any cryptographic export license concerns.
>>
>> == Required Resources ==
>> === Mailing Lists ===
>>    * user@nifi.incubator.apache.org
>>    * dev@nifi.incubator.apache.org
>>    * private@nifi.incubator.apache.org
>>    * commits@nifi.incubator.apache.org
>>
>> === Source Control ===
>> NiFi requests use of Git for source control
>> (git://git.apache.org/nifi.git).  We request a writeable Git repo for
>> NiFi with mirroring to be setup to Github through INFRA.  We request
>> sponsor Benson Margulies (bimargulies) to assist with creating the
>> INFRA ticket for this.
>>
>> === Issue Tracking ===
>> JIRA NiFi (NIFI)
>>
>> === Initial Committers ===
>>    * Brandon Devries <brandon.devries at gmail dot com>
>>    * Matt Gilman <matt.c.gilman at gmail dot com>
>>    * Tony Kurc <trkurc at gmail dot com>
>>    * Mark Payne <markap14 at hotmail dot com>
>>    * Adam Taft <adam at adamtaft dot com>
>>    * Joseph Witt <joewitt at gmail dot com>
>>
>> === Affiliations ===
>>    * Brandon Devries (Requitest, Inc.)
>>    * Matt Gilman (Raytheon)
>>    * Tony Kurc (National Security Agency)
>>    * Mark Payne (Sotera Defense Solutions, Inc.)
>>    * Adam Taft (Requitest, Inc.)
>>    * Joseph Witt (National Security Agency)
>>
>> == Sponsors ==
>> === Champion ===
>>    * Benson Margulies (Basis Technology) <bimargulies at apache dot org>
>>
>> === Nominated Mentors ===
>>    * Drew Farris (Booz Allen Hamilton) <drew at apache dot org>
>>    * Brock Noland (Cloudera) <brock at apache dot org>
>>    * Billie Rinaldi (Hortonworks) <billie at apache dot org>
>>
>> === Sponsoring Entity ===
>> We request the Apache Incubator to sponsor this project.
>>
>>
> --
> Sergio Fernández
> Senior Researcher
> Knowledge and Media Technologies
> Salzburg Research Forschungsgesellschaft mbH
> Jakob-Haringer-Straße 5/3 | 5020 Salzburg, Austria
> T: +43 662 2288 318 | M: +43 660 2747 925
> sergio.fernandez@salzburgresearch.at
> http://www.salzburgresearch.at
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [PROPOSAL] NiFi for Incubation

[Sergio Fernández <se...@salzburgresearch.at>]

Very interesting project!
I'm happy to help mentoring it.

On 20/11/14 03:02, Joe Witt wrote:
> Hello,
>
>
> I would like to propose NiFi as an Apache Incubator Project.
>
> In addition to the copy provided below the Wiki version of the
> proposal can be found here:
> http://wiki.apache.org/incubator/NiFiProposal
>
> Thanks
>
> Joe
>
>
> = NiFi Proposal =
>
> == Abstract ==
> NiFi is a dataflow system based on the concepts of flow-based programming.
>
> == Proposal ==
> NiFi supports powerful and scalable directed graphs of data routing,
> transformation, and system mediation logic.  Some of the high-level
> capabilities and objectives of NiFi include:
>    * Web-based user interface for seamless experience between design,
> control, feedback, and monitoring of data flows
>    * Highly configurable along several dimensions of quality of service
> such as loss tolerant versus guaranteed delivery, low latency versus
> high throughput, and priority based queuing
>    * Fine-grained data provenance for all data received, forked,
> joined, cloned, modified, sent, and ultimately dropped as data reaches
> its configured end-state
>    * Component-based extension model along well defined interfaces
> enabling rapid development and effective testing
>
> == Background ==
> Reliable and effective dataflow between systems can be difficult
> whether you're running scripts on a laptop or have a massive
> distributed computing system operated by numerous teams and
> organizations.  As the volume and rate of data grows and as the number
> of systems, protocols, and formats increase and evolve so too does the
> complexity and need for greater insight and agility.  These are the
> dataflow challenges that NiFi was built to tackle.
>
> NiFi is designed in a manner consistent with the core concepts
> described in flow-based programming as originally documented by J.
> Paul Morrison in the 1970s.  This model lends itself well to visual
> diagramming, concurrency, componentization, testing, and reuse.  In
> addition to staying close to the fundamentals of flow-based
> programming, NiFi provides integration system specific features such
> as: guaranteed delivery; back pressure; ability to gracefully handle
> backlogs and data surges; and an operator interface that enables
> on-the-fly data flow generation, modification, and observation.
>
> == Rationale ==
> NiFi provides a reliable, scalable, manageable and accountable
> platform for developers and technical staff to create and evolve
> powerful data flows.  Such a system is useful in many contexts
> including large-scale enterprise integration, interaction with cloud
> services and frameworks, business to business, intra-departmental, and
> inter-departmental flows.  NiFi fits well within the Apache Software
> Foundation (ASF) family as it depends on numerous ASF projects and
> integrates with several others.  We also anticipate developing
> extensions for several other ASF projects such as Cassandra, Kafka,
> and Storm in the near future.
>
> == Initial Goals ==
>    * Ensure all dependencies are compliant with Apache License version
> 2.0 and all that all code and documentation artifacts have the correct
> Apache licensing markings and notice.
>    * Establish a formal release process and schedule, allowing for
> dependable release cycles in a manner consistent with the Apache
> development process.
>    * Determine and establish a mechanism, possibly including a
> sub-project construct, that allows for extensions to the core
> application to occur at a pace that differs from the core application
> itself.
>
> == Current Status ==
> === Meritocracy ===
> An integration platform is only as good as its ability to integrate
> systems in a reliable, timely, and repeatable manner.  The same can be
> said of its ability to attract talent and a variety of perspectives as
> integration systems by their nature are always evolving.  We will
> actively seek help and encourage promotion of influence in the project
> through meritocracy.
>
> === Community ===
> Over the past several years, NiFi has developed a strong community of
> both developers and operators within the U.S. government.  We look
> forward to helping grow this to a broader base of industries.
>
> === Core Developers ===
> The initial core developers are employed by the National Security
> Agency and defense contractors.  We will work to grow the community
> among a more diverse set of developers and industries.
>
> === Alignment ===
>  From its inception, NiFi was developed with an open source philosophy
> in mind and with the hopes of eventually being truly open sourced.
> The Apache way is consistent with the approach we have taken to date.
> The ASF clearly provides a mature and effective environment for
> successful development as is evident across the spectrum of well-known
> projects.  Further, NiFi depends on numerous ASF libraries and
> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
> HttpClient, Jakarta and Maven.  We also anticipate extensions and
> dependencies with several more ASF projects, including Accumulo, Avro,
> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
> This existing alignment with Apache and the desired community makes
> the Apache Incubator a good fit for NiFi.
>
> == Known Risks ==
> === Orphaned Products ===
> Risk of orphaning is minimal.  The project user and developer base is
> substantial, growing, and there is already extensive operational use
> of NiFi.
>
> === Inexperience with Open Source ===
> The initial committers to NiFi have limited experience with true open
> source software development.  However, despite the project origins
> being from closed source development we have modelled our behavior and
> community development on The Apache Way to the greatest extent
> possible.  This environment includes widely accessible source code
> repositories, published artifacts, ticket tracking, and extensive
> documentation. We also encourage contributions and frequent debate and
> hold regular, collaborative discussions through e-mail, chat rooms,
> and in-person meet-ups.  We are committed to the ideals of open source
> software and will eagerly seek out mentors and sponsors who can help
> us quickly come up to speed.
>
> === Homogenous Developers ===
> The initial committers of NiFi come from a limited set of entities
> though we are committed to recruiting and developing additional
> committers from a broad spectrum of industries and backgrounds.
>
> === Reliance on Salaried Developers ===
> We expect NiFi development to continue on salaried time and through
> volunteer time.  The initial committers are paid by their employers to
> contribute to this project.  We are committed to developing and
> recruiting participation from developers both salaried and
> non-salaried.
>
> === Relationship with other Apache Projects ===
> As described in the alignment section, NiFi is already heavily
> dependent on other ASF projects and we anticipate further dependence
> and integration with new and emerging projects in the Apache family.
>
> === An Excessive Fascination with the Apache Brand ===
> We respect the laudable Apache brand and that is certainly a factor in
> the decision to propose NiFi for the Apache Incubator.  We intend to
> provide a great deal of energy and capability to the ASF through this
> project.  We will be sensitive to and respectful of any overuse of the
> Apache brand and ensure our focus remains on how we benefit the Apache
> community.
>
> === Documentation ===
> At this time there is no NiFi documentation on the web.  However, we
> have extensive documentation included within the application that
> details usage of the many functions.  We will be rapidly expanding the
> available documentation to cover things like installation, developer
> guide, frequently asked questions, best practices, and more.  This
> documentation will be posted to the NiFi wiki at apache.org.
>
> == Initial Source ==
> NiFi has been in active development since late 2006 with contributions
> from dozens of developers and feedback from hundreds of users and
> developers.  The core codebase is written in Java and includes
> detailed Javadocs and feature documentation.
>
> == Source and Intellectual Property Submission ==
> Previously referred to as Niagarafiles, the NiFi code and
> documentation materials will be submitted by the National Security
> Agency.  NiFi has been developed by a mix of government employees and
> private companies under government contract.  Material developed by
> the government employees is in the public domain and no U.S. copyright
> exists in works of the federal government.  For the contractor
> developed material in the initial submission, the U.S. Government has
> sufficient authority to open source per DFARS 252.227-7014.  NSA has
> submitted the Software Grant Agreement and Corporate Contributor
> License Agreement to the Apache Software Foundation.
>
> == External Dependencies ==
> We have at least one dependency on an LGPL library which we will
> promptly address.  Otherwise, we believe all current dependencies are
> compatible with the ASF guidelines.  Our dependency licenses come from
> the following license styles:  Apache v 2.0, BSD, Public Domain,
> Eclipse Public v1, MIT, CDDL v1.
>
> == Cryptography ==
> Consistent with http://www.apache.org/licenses/exports/ we believe
> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
> cryptographic algorithms but is designed to use algorithms provided by
> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
> These cryptographic algorithm providers are used to support SSL,
> SSH/SFTP, and the encryption and decryption of sensitive properties.
> In the event that it becomes necessary we will engage with appropriate
> Apache members to ensure we file any necessary paperwork or clarified
> any cryptographic export license concerns.
>
> == Required Resources ==
> === Mailing Lists ===
>    * user@nifi.incubator.apache.org
>    * dev@nifi.incubator.apache.org
>    * private@nifi.incubator.apache.org
>    * commits@nifi.incubator.apache.org
>
> === Source Control ===
> NiFi requests use of Git for source control
> (git://git.apache.org/nifi.git).  We request a writeable Git repo for
> NiFi with mirroring to be setup to Github through INFRA.  We request
> sponsor Benson Margulies (bimargulies) to assist with creating the
> INFRA ticket for this.
>
> === Issue Tracking ===
> JIRA NiFi (NIFI)
>
> === Initial Committers ===
>    * Brandon Devries <brandon.devries at gmail dot com>
>    * Matt Gilman <matt.c.gilman at gmail dot com>
>    * Tony Kurc <trkurc at gmail dot com>
>    * Mark Payne <markap14 at hotmail dot com>
>    * Adam Taft <adam at adamtaft dot com>
>    * Joseph Witt <joewitt at gmail dot com>
>
> === Affiliations ===
>    * Brandon Devries (Requitest, Inc.)
>    * Matt Gilman (Raytheon)
>    * Tony Kurc (National Security Agency)
>    * Mark Payne (Sotera Defense Solutions, Inc.)
>    * Adam Taft (Requitest, Inc.)
>    * Joseph Witt (National Security Agency)
>
> == Sponsors ==
> === Champion ===
>    * Benson Margulies (Basis Technology) <bimargulies at apache dot org>
>
> === Nominated Mentors ===
>    * Drew Farris (Booz Allen Hamilton) <drew at apache dot org>
>    * Brock Noland (Cloudera) <brock at apache dot org>
>    * Billie Rinaldi (Hortonworks) <billie at apache dot org>
>
> === Sponsoring Entity ===
> We request the Apache Incubator to sponsor this project.
>

-- 
Sergio Fernández
Senior Researcher
Knowledge and Media Technologies
Salzburg Research Forschungsgesellschaft mbH
Jakob-Haringer-Straße 5/3 | 5020 Salzburg, Austria
T: +43 662 2288 318 | M: +43 660 2747 925
sergio.fernandez@salzburgresearch.at
http://www.salzburgresearch.at

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [PROPOSAL] NiFi for Incubation

[Joe Witt <jo...@gmail.com>]

Arvind

Thank you for offering to be a mentor.  The proposal has been updated.

Arvind Prabhakar (StreamSets) <arvind at apache dot org>

One of the comments we have received thus far is a potential concern of
whether multiple communities/projects can exist in similar problem areas.
Your background in Flume which solves similar problems to NiFi is a good
sign that multiple projects and communities can co-exist within the space
and even provide complimentary capability.

Thanks
Joe

On Thu, Nov 20, 2014 at 9:20 PM, Joe Witt <jo...@gmail.com> wrote:

> Josh,
>
> Really appreciate it and have updated the proposal.
>
> Thanks
> Joe
>
> On Thu, Nov 20, 2014 at 9:35 AM, Josh Elser <el...@apache.org> wrote:
>
>> Very exciting stuff!
>>
>> Not presently on IPMC, but if you'd have me, I'd be happy to volunteer as
>> a mentor. If so, I'll submit an application to join the IPMC and we can go
>> from there.
>>
>> - Josh
>>
>>
>> Joe Witt wrote:
>>
>>> Hello,
>>>
>>>
>>> I would like to propose NiFi as an Apache Incubator Project.
>>>
>>> In addition to the copy provided below the Wiki version of the
>>> proposal can be found here:
>>> http://wiki.apache.org/incubator/NiFiProposal
>>>
>>> Thanks
>>>
>>> Joe
>>>
>>>
>>> = NiFi Proposal =
>>>
>>> == Abstract ==
>>> NiFi is a dataflow system based on the concepts of flow-based
>>> programming.
>>>
>>> == Proposal ==
>>> NiFi supports powerful and scalable directed graphs of data routing,
>>> transformation, and system mediation logic.  Some of the high-level
>>> capabilities and objectives of NiFi include:
>>>    * Web-based user interface for seamless experience between design,
>>> control, feedback, and monitoring of data flows
>>>    * Highly configurable along several dimensions of quality of service
>>> such as loss tolerant versus guaranteed delivery, low latency versus
>>> high throughput, and priority based queuing
>>>    * Fine-grained data provenance for all data received, forked,
>>> joined, cloned, modified, sent, and ultimately dropped as data reaches
>>> its configured end-state
>>>    * Component-based extension model along well defined interfaces
>>> enabling rapid development and effective testing
>>>
>>> == Background ==
>>> Reliable and effective dataflow between systems can be difficult
>>> whether you're running scripts on a laptop or have a massive
>>> distributed computing system operated by numerous teams and
>>> organizations.  As the volume and rate of data grows and as the number
>>> of systems, protocols, and formats increase and evolve so too does the
>>> complexity and need for greater insight and agility.  These are the
>>> dataflow challenges that NiFi was built to tackle.
>>>
>>> NiFi is designed in a manner consistent with the core concepts
>>> described in flow-based programming as originally documented by J.
>>> Paul Morrison in the 1970s.  This model lends itself well to visual
>>> diagramming, concurrency, componentization, testing, and reuse.  In
>>> addition to staying close to the fundamentals of flow-based
>>> programming, NiFi provides integration system specific features such
>>> as: guaranteed delivery; back pressure; ability to gracefully handle
>>> backlogs and data surges; and an operator interface that enables
>>> on-the-fly data flow generation, modification, and observation.
>>>
>>> == Rationale ==
>>> NiFi provides a reliable, scalable, manageable and accountable
>>> platform for developers and technical staff to create and evolve
>>> powerful data flows.  Such a system is useful in many contexts
>>> including large-scale enterprise integration, interaction with cloud
>>> services and frameworks, business to business, intra-departmental, and
>>> inter-departmental flows.  NiFi fits well within the Apache Software
>>> Foundation (ASF) family as it depends on numerous ASF projects and
>>> integrates with several others.  We also anticipate developing
>>> extensions for several other ASF projects such as Cassandra, Kafka,
>>> and Storm in the near future.
>>>
>>> == Initial Goals ==
>>>    * Ensure all dependencies are compliant with Apache License version
>>> 2.0 and all that all code and documentation artifacts have the correct
>>> Apache licensing markings and notice.
>>>    * Establish a formal release process and schedule, allowing for
>>> dependable release cycles in a manner consistent with the Apache
>>> development process.
>>>    * Determine and establish a mechanism, possibly including a
>>> sub-project construct, that allows for extensions to the core
>>> application to occur at a pace that differs from the core application
>>> itself.
>>>
>>> == Current Status ==
>>> === Meritocracy ===
>>> An integration platform is only as good as its ability to integrate
>>> systems in a reliable, timely, and repeatable manner.  The same can be
>>> said of its ability to attract talent and a variety of perspectives as
>>> integration systems by their nature are always evolving.  We will
>>> actively seek help and encourage promotion of influence in the project
>>> through meritocracy.
>>>
>>> === Community ===
>>> Over the past several years, NiFi has developed a strong community of
>>> both developers and operators within the U.S. government.  We look
>>> forward to helping grow this to a broader base of industries.
>>>
>>> === Core Developers ===
>>> The initial core developers are employed by the National Security
>>> Agency and defense contractors.  We will work to grow the community
>>> among a more diverse set of developers and industries.
>>>
>>> === Alignment ===
>>>  From its inception, NiFi was developed with an open source philosophy
>>> in mind and with the hopes of eventually being truly open sourced.
>>> The Apache way is consistent with the approach we have taken to date.
>>> The ASF clearly provides a mature and effective environment for
>>> successful development as is evident across the spectrum of well-known
>>> projects.  Further, NiFi depends on numerous ASF libraries and
>>> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
>>> HttpClient, Jakarta and Maven.  We also anticipate extensions and
>>> dependencies with several more ASF projects, including Accumulo, Avro,
>>> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
>>> This existing alignment with Apache and the desired community makes
>>> the Apache Incubator a good fit for NiFi.
>>>
>>> == Known Risks ==
>>> === Orphaned Products ===
>>> Risk of orphaning is minimal.  The project user and developer base is
>>> substantial, growing, and there is already extensive operational use
>>> of NiFi.
>>>
>>> === Inexperience with Open Source ===
>>> The initial committers to NiFi have limited experience with true open
>>> source software development.  However, despite the project origins
>>> being from closed source development we have modelled our behavior and
>>> community development on The Apache Way to the greatest extent
>>> possible.  This environment includes widely accessible source code
>>> repositories, published artifacts, ticket tracking, and extensive
>>> documentation. We also encourage contributions and frequent debate and
>>> hold regular, collaborative discussions through e-mail, chat rooms,
>>> and in-person meet-ups.  We are committed to the ideals of open source
>>> software and will eagerly seek out mentors and sponsors who can help
>>> us quickly come up to speed.
>>>
>>> === Homogenous Developers ===
>>> The initial committers of NiFi come from a limited set of entities
>>> though we are committed to recruiting and developing additional
>>> committers from a broad spectrum of industries and backgrounds.
>>>
>>> === Reliance on Salaried Developers ===
>>> We expect NiFi development to continue on salaried time and through
>>> volunteer time.  The initial committers are paid by their employers to
>>> contribute to this project.  We are committed to developing and
>>> recruiting participation from developers both salaried and
>>> non-salaried.
>>>
>>> === Relationship with other Apache Projects ===
>>> As described in the alignment section, NiFi is already heavily
>>> dependent on other ASF projects and we anticipate further dependence
>>> and integration with new and emerging projects in the Apache family.
>>>
>>> === An Excessive Fascination with the Apache Brand ===
>>> We respect the laudable Apache brand and that is certainly a factor in
>>> the decision to propose NiFi for the Apache Incubator.  We intend to
>>> provide a great deal of energy and capability to the ASF through this
>>> project.  We will be sensitive to and respectful of any overuse of the
>>> Apache brand and ensure our focus remains on how we benefit the Apache
>>> community.
>>>
>>> === Documentation ===
>>> At this time there is no NiFi documentation on the web.  However, we
>>> have extensive documentation included within the application that
>>> details usage of the many functions.  We will be rapidly expanding the
>>> available documentation to cover things like installation, developer
>>> guide, frequently asked questions, best practices, and more.  This
>>> documentation will be posted to the NiFi wiki at apache.org.
>>>
>>> == Initial Source ==
>>> NiFi has been in active development since late 2006 with contributions
>>> from dozens of developers and feedback from hundreds of users and
>>> developers.  The core codebase is written in Java and includes
>>> detailed Javadocs and feature documentation.
>>>
>>> == Source and Intellectual Property Submission ==
>>> Previously referred to as Niagarafiles, the NiFi code and
>>> documentation materials will be submitted by the National Security
>>> Agency.  NiFi has been developed by a mix of government employees and
>>> private companies under government contract.  Material developed by
>>> the government employees is in the public domain and no U.S. copyright
>>> exists in works of the federal government.  For the contractor
>>> developed material in the initial submission, the U.S. Government has
>>> sufficient authority to open source per DFARS 252.227-7014.  NSA has
>>> submitted the Software Grant Agreement and Corporate Contributor
>>> License Agreement to the Apache Software Foundation.
>>>
>>> == External Dependencies ==
>>> We have at least one dependency on an LGPL library which we will
>>> promptly address.  Otherwise, we believe all current dependencies are
>>> compatible with the ASF guidelines.  Our dependency licenses come from
>>> the following license styles:  Apache v 2.0, BSD, Public Domain,
>>> Eclipse Public v1, MIT, CDDL v1.
>>>
>>> == Cryptography ==
>>> Consistent with http://www.apache.org/licenses/exports/ we believe
>>> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
>>> cryptographic algorithms but is designed to use algorithms provided by
>>> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
>>> These cryptographic algorithm providers are used to support SSL,
>>> SSH/SFTP, and the encryption and decryption of sensitive properties.
>>> In the event that it becomes necessary we will engage with appropriate
>>> Apache members to ensure we file any necessary paperwork or clarified
>>> any cryptographic export license concerns.
>>>
>>> == Required Resources ==
>>> === Mailing Lists ===
>>>    * user@nifi.incubator.apache.org
>>>    * dev@nifi.incubator.apache.org
>>>    * private@nifi.incubator.apache.org
>>>    * commits@nifi.incubator.apache.org
>>>
>>> === Source Control ===
>>> NiFi requests use of Git for source control
>>> (git://git.apache.org/nifi.git).  We request a writeable Git repo for
>>> NiFi with mirroring to be setup to Github through INFRA.  We request
>>> sponsor Benson Margulies (bimargulies) to assist with creating the
>>> INFRA ticket for this.
>>>
>>> === Issue Tracking ===
>>> JIRA NiFi (NIFI)
>>>
>>> === Initial Committers ===
>>>    * Brandon Devries<brandon.devries at gmail dot com>
>>>    * Matt Gilman<matt.c.gilman at gmail dot com>
>>>    * Tony Kurc<trkurc at gmail dot com>
>>>    * Mark Payne<markap14 at hotmail dot com>
>>>    * Adam Taft<adam at adamtaft dot com>
>>>    * Joseph Witt<joewitt at gmail dot com>
>>>
>>> === Affiliations ===
>>>    * Brandon Devries (Requitest, Inc.)
>>>    * Matt Gilman (Raytheon)
>>>    * Tony Kurc (National Security Agency)
>>>    * Mark Payne (Sotera Defense Solutions, Inc.)
>>>    * Adam Taft (Requitest, Inc.)
>>>    * Joseph Witt (National Security Agency)
>>>
>>> == Sponsors ==
>>> === Champion ===
>>>    * Benson Margulies (Basis Technology)<bimargulies at apache dot org>
>>>
>>> === Nominated Mentors ===
>>>    * Drew Farris (Booz Allen Hamilton)<drew at apache dot org>
>>>    * Brock Noland (Cloudera)<brock at apache dot org>
>>>    * Billie Rinaldi (Hortonworks)<billie at apache dot org>
>>>
>>> === Sponsoring Entity ===
>>> We request the Apache Incubator to sponsor this project.
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
>

Re: [PROPOSAL] NiFi for Incubation

[Joe Witt <jo...@gmail.com>]

Josh,

Really appreciate it and have updated the proposal.

Thanks
Joe

On Thu, Nov 20, 2014 at 9:35 AM, Josh Elser <el...@apache.org> wrote:

> Very exciting stuff!
>
> Not presently on IPMC, but if you'd have me, I'd be happy to volunteer as
> a mentor. If so, I'll submit an application to join the IPMC and we can go
> from there.
>
> - Josh
>
>
> Joe Witt wrote:
>
>> Hello,
>>
>>
>> I would like to propose NiFi as an Apache Incubator Project.
>>
>> In addition to the copy provided below the Wiki version of the
>> proposal can be found here:
>> http://wiki.apache.org/incubator/NiFiProposal
>>
>> Thanks
>>
>> Joe
>>
>>
>> = NiFi Proposal =
>>
>> == Abstract ==
>> NiFi is a dataflow system based on the concepts of flow-based programming.
>>
>> == Proposal ==
>> NiFi supports powerful and scalable directed graphs of data routing,
>> transformation, and system mediation logic.  Some of the high-level
>> capabilities and objectives of NiFi include:
>>    * Web-based user interface for seamless experience between design,
>> control, feedback, and monitoring of data flows
>>    * Highly configurable along several dimensions of quality of service
>> such as loss tolerant versus guaranteed delivery, low latency versus
>> high throughput, and priority based queuing
>>    * Fine-grained data provenance for all data received, forked,
>> joined, cloned, modified, sent, and ultimately dropped as data reaches
>> its configured end-state
>>    * Component-based extension model along well defined interfaces
>> enabling rapid development and effective testing
>>
>> == Background ==
>> Reliable and effective dataflow between systems can be difficult
>> whether you're running scripts on a laptop or have a massive
>> distributed computing system operated by numerous teams and
>> organizations.  As the volume and rate of data grows and as the number
>> of systems, protocols, and formats increase and evolve so too does the
>> complexity and need for greater insight and agility.  These are the
>> dataflow challenges that NiFi was built to tackle.
>>
>> NiFi is designed in a manner consistent with the core concepts
>> described in flow-based programming as originally documented by J.
>> Paul Morrison in the 1970s.  This model lends itself well to visual
>> diagramming, concurrency, componentization, testing, and reuse.  In
>> addition to staying close to the fundamentals of flow-based
>> programming, NiFi provides integration system specific features such
>> as: guaranteed delivery; back pressure; ability to gracefully handle
>> backlogs and data surges; and an operator interface that enables
>> on-the-fly data flow generation, modification, and observation.
>>
>> == Rationale ==
>> NiFi provides a reliable, scalable, manageable and accountable
>> platform for developers and technical staff to create and evolve
>> powerful data flows.  Such a system is useful in many contexts
>> including large-scale enterprise integration, interaction with cloud
>> services and frameworks, business to business, intra-departmental, and
>> inter-departmental flows.  NiFi fits well within the Apache Software
>> Foundation (ASF) family as it depends on numerous ASF projects and
>> integrates with several others.  We also anticipate developing
>> extensions for several other ASF projects such as Cassandra, Kafka,
>> and Storm in the near future.
>>
>> == Initial Goals ==
>>    * Ensure all dependencies are compliant with Apache License version
>> 2.0 and all that all code and documentation artifacts have the correct
>> Apache licensing markings and notice.
>>    * Establish a formal release process and schedule, allowing for
>> dependable release cycles in a manner consistent with the Apache
>> development process.
>>    * Determine and establish a mechanism, possibly including a
>> sub-project construct, that allows for extensions to the core
>> application to occur at a pace that differs from the core application
>> itself.
>>
>> == Current Status ==
>> === Meritocracy ===
>> An integration platform is only as good as its ability to integrate
>> systems in a reliable, timely, and repeatable manner.  The same can be
>> said of its ability to attract talent and a variety of perspectives as
>> integration systems by their nature are always evolving.  We will
>> actively seek help and encourage promotion of influence in the project
>> through meritocracy.
>>
>> === Community ===
>> Over the past several years, NiFi has developed a strong community of
>> both developers and operators within the U.S. government.  We look
>> forward to helping grow this to a broader base of industries.
>>
>> === Core Developers ===
>> The initial core developers are employed by the National Security
>> Agency and defense contractors.  We will work to grow the community
>> among a more diverse set of developers and industries.
>>
>> === Alignment ===
>>  From its inception, NiFi was developed with an open source philosophy
>> in mind and with the hopes of eventually being truly open sourced.
>> The Apache way is consistent with the approach we have taken to date.
>> The ASF clearly provides a mature and effective environment for
>> successful development as is evident across the spectrum of well-known
>> projects.  Further, NiFi depends on numerous ASF libraries and
>> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
>> HttpClient, Jakarta and Maven.  We also anticipate extensions and
>> dependencies with several more ASF projects, including Accumulo, Avro,
>> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
>> This existing alignment with Apache and the desired community makes
>> the Apache Incubator a good fit for NiFi.
>>
>> == Known Risks ==
>> === Orphaned Products ===
>> Risk of orphaning is minimal.  The project user and developer base is
>> substantial, growing, and there is already extensive operational use
>> of NiFi.
>>
>> === Inexperience with Open Source ===
>> The initial committers to NiFi have limited experience with true open
>> source software development.  However, despite the project origins
>> being from closed source development we have modelled our behavior and
>> community development on The Apache Way to the greatest extent
>> possible.  This environment includes widely accessible source code
>> repositories, published artifacts, ticket tracking, and extensive
>> documentation. We also encourage contributions and frequent debate and
>> hold regular, collaborative discussions through e-mail, chat rooms,
>> and in-person meet-ups.  We are committed to the ideals of open source
>> software and will eagerly seek out mentors and sponsors who can help
>> us quickly come up to speed.
>>
>> === Homogenous Developers ===
>> The initial committers of NiFi come from a limited set of entities
>> though we are committed to recruiting and developing additional
>> committers from a broad spectrum of industries and backgrounds.
>>
>> === Reliance on Salaried Developers ===
>> We expect NiFi development to continue on salaried time and through
>> volunteer time.  The initial committers are paid by their employers to
>> contribute to this project.  We are committed to developing and
>> recruiting participation from developers both salaried and
>> non-salaried.
>>
>> === Relationship with other Apache Projects ===
>> As described in the alignment section, NiFi is already heavily
>> dependent on other ASF projects and we anticipate further dependence
>> and integration with new and emerging projects in the Apache family.
>>
>> === An Excessive Fascination with the Apache Brand ===
>> We respect the laudable Apache brand and that is certainly a factor in
>> the decision to propose NiFi for the Apache Incubator.  We intend to
>> provide a great deal of energy and capability to the ASF through this
>> project.  We will be sensitive to and respectful of any overuse of the
>> Apache brand and ensure our focus remains on how we benefit the Apache
>> community.
>>
>> === Documentation ===
>> At this time there is no NiFi documentation on the web.  However, we
>> have extensive documentation included within the application that
>> details usage of the many functions.  We will be rapidly expanding the
>> available documentation to cover things like installation, developer
>> guide, frequently asked questions, best practices, and more.  This
>> documentation will be posted to the NiFi wiki at apache.org.
>>
>> == Initial Source ==
>> NiFi has been in active development since late 2006 with contributions
>> from dozens of developers and feedback from hundreds of users and
>> developers.  The core codebase is written in Java and includes
>> detailed Javadocs and feature documentation.
>>
>> == Source and Intellectual Property Submission ==
>> Previously referred to as Niagarafiles, the NiFi code and
>> documentation materials will be submitted by the National Security
>> Agency.  NiFi has been developed by a mix of government employees and
>> private companies under government contract.  Material developed by
>> the government employees is in the public domain and no U.S. copyright
>> exists in works of the federal government.  For the contractor
>> developed material in the initial submission, the U.S. Government has
>> sufficient authority to open source per DFARS 252.227-7014.  NSA has
>> submitted the Software Grant Agreement and Corporate Contributor
>> License Agreement to the Apache Software Foundation.
>>
>> == External Dependencies ==
>> We have at least one dependency on an LGPL library which we will
>> promptly address.  Otherwise, we believe all current dependencies are
>> compatible with the ASF guidelines.  Our dependency licenses come from
>> the following license styles:  Apache v 2.0, BSD, Public Domain,
>> Eclipse Public v1, MIT, CDDL v1.
>>
>> == Cryptography ==
>> Consistent with http://www.apache.org/licenses/exports/ we believe
>> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
>> cryptographic algorithms but is designed to use algorithms provided by
>> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
>> These cryptographic algorithm providers are used to support SSL,
>> SSH/SFTP, and the encryption and decryption of sensitive properties.
>> In the event that it becomes necessary we will engage with appropriate
>> Apache members to ensure we file any necessary paperwork or clarified
>> any cryptographic export license concerns.
>>
>> == Required Resources ==
>> === Mailing Lists ===
>>    * user@nifi.incubator.apache.org
>>    * dev@nifi.incubator.apache.org
>>    * private@nifi.incubator.apache.org
>>    * commits@nifi.incubator.apache.org
>>
>> === Source Control ===
>> NiFi requests use of Git for source control
>> (git://git.apache.org/nifi.git).  We request a writeable Git repo for
>> NiFi with mirroring to be setup to Github through INFRA.  We request
>> sponsor Benson Margulies (bimargulies) to assist with creating the
>> INFRA ticket for this.
>>
>> === Issue Tracking ===
>> JIRA NiFi (NIFI)
>>
>> === Initial Committers ===
>>    * Brandon Devries<brandon.devries at gmail dot com>
>>    * Matt Gilman<matt.c.gilman at gmail dot com>
>>    * Tony Kurc<trkurc at gmail dot com>
>>    * Mark Payne<markap14 at hotmail dot com>
>>    * Adam Taft<adam at adamtaft dot com>
>>    * Joseph Witt<joewitt at gmail dot com>
>>
>> === Affiliations ===
>>    * Brandon Devries (Requitest, Inc.)
>>    * Matt Gilman (Raytheon)
>>    * Tony Kurc (National Security Agency)
>>    * Mark Payne (Sotera Defense Solutions, Inc.)
>>    * Adam Taft (Requitest, Inc.)
>>    * Joseph Witt (National Security Agency)
>>
>> == Sponsors ==
>> === Champion ===
>>    * Benson Margulies (Basis Technology)<bimargulies at apache dot org>
>>
>> === Nominated Mentors ===
>>    * Drew Farris (Booz Allen Hamilton)<drew at apache dot org>
>>    * Brock Noland (Cloudera)<brock at apache dot org>
>>    * Billie Rinaldi (Hortonworks)<billie at apache dot org>
>>
>> === Sponsoring Entity ===
>> We request the Apache Incubator to sponsor this project.
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [PROPOSAL] NiFi for Incubation

[Josh Elser <el...@apache.org>]

Very exciting stuff!

Not presently on IPMC, but if you'd have me, I'd be happy to volunteer 
as a mentor. If so, I'll submit an application to join the IPMC and we 
can go from there.

- Josh

Joe Witt wrote:
> Hello,
>
>
> I would like to propose NiFi as an Apache Incubator Project.
>
> In addition to the copy provided below the Wiki version of the
> proposal can be found here:
> http://wiki.apache.org/incubator/NiFiProposal
>
> Thanks
>
> Joe
>
>
> = NiFi Proposal =
>
> == Abstract ==
> NiFi is a dataflow system based on the concepts of flow-based programming.
>
> == Proposal ==
> NiFi supports powerful and scalable directed graphs of data routing,
> transformation, and system mediation logic.  Some of the high-level
> capabilities and objectives of NiFi include:
>    * Web-based user interface for seamless experience between design,
> control, feedback, and monitoring of data flows
>    * Highly configurable along several dimensions of quality of service
> such as loss tolerant versus guaranteed delivery, low latency versus
> high throughput, and priority based queuing
>    * Fine-grained data provenance for all data received, forked,
> joined, cloned, modified, sent, and ultimately dropped as data reaches
> its configured end-state
>    * Component-based extension model along well defined interfaces
> enabling rapid development and effective testing
>
> == Background ==
> Reliable and effective dataflow between systems can be difficult
> whether you're running scripts on a laptop or have a massive
> distributed computing system operated by numerous teams and
> organizations.  As the volume and rate of data grows and as the number
> of systems, protocols, and formats increase and evolve so too does the
> complexity and need for greater insight and agility.  These are the
> dataflow challenges that NiFi was built to tackle.
>
> NiFi is designed in a manner consistent with the core concepts
> described in flow-based programming as originally documented by J.
> Paul Morrison in the 1970s.  This model lends itself well to visual
> diagramming, concurrency, componentization, testing, and reuse.  In
> addition to staying close to the fundamentals of flow-based
> programming, NiFi provides integration system specific features such
> as: guaranteed delivery; back pressure; ability to gracefully handle
> backlogs and data surges; and an operator interface that enables
> on-the-fly data flow generation, modification, and observation.
>
> == Rationale ==
> NiFi provides a reliable, scalable, manageable and accountable
> platform for developers and technical staff to create and evolve
> powerful data flows.  Such a system is useful in many contexts
> including large-scale enterprise integration, interaction with cloud
> services and frameworks, business to business, intra-departmental, and
> inter-departmental flows.  NiFi fits well within the Apache Software
> Foundation (ASF) family as it depends on numerous ASF projects and
> integrates with several others.  We also anticipate developing
> extensions for several other ASF projects such as Cassandra, Kafka,
> and Storm in the near future.
>
> == Initial Goals ==
>    * Ensure all dependencies are compliant with Apache License version
> 2.0 and all that all code and documentation artifacts have the correct
> Apache licensing markings and notice.
>    * Establish a formal release process and schedule, allowing for
> dependable release cycles in a manner consistent with the Apache
> development process.
>    * Determine and establish a mechanism, possibly including a
> sub-project construct, that allows for extensions to the core
> application to occur at a pace that differs from the core application
> itself.
>
> == Current Status ==
> === Meritocracy ===
> An integration platform is only as good as its ability to integrate
> systems in a reliable, timely, and repeatable manner.  The same can be
> said of its ability to attract talent and a variety of perspectives as
> integration systems by their nature are always evolving.  We will
> actively seek help and encourage promotion of influence in the project
> through meritocracy.
>
> === Community ===
> Over the past several years, NiFi has developed a strong community of
> both developers and operators within the U.S. government.  We look
> forward to helping grow this to a broader base of industries.
>
> === Core Developers ===
> The initial core developers are employed by the National Security
> Agency and defense contractors.  We will work to grow the community
> among a more diverse set of developers and industries.
>
> === Alignment ===
>  From its inception, NiFi was developed with an open source philosophy
> in mind and with the hopes of eventually being truly open sourced.
> The Apache way is consistent with the approach we have taken to date.
> The ASF clearly provides a mature and effective environment for
> successful development as is evident across the spectrum of well-known
> projects.  Further, NiFi depends on numerous ASF libraries and
> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
> HttpClient, Jakarta and Maven.  We also anticipate extensions and
> dependencies with several more ASF projects, including Accumulo, Avro,
> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
> This existing alignment with Apache and the desired community makes
> the Apache Incubator a good fit for NiFi.
>
> == Known Risks ==
> === Orphaned Products ===
> Risk of orphaning is minimal.  The project user and developer base is
> substantial, growing, and there is already extensive operational use
> of NiFi.
>
> === Inexperience with Open Source ===
> The initial committers to NiFi have limited experience with true open
> source software development.  However, despite the project origins
> being from closed source development we have modelled our behavior and
> community development on The Apache Way to the greatest extent
> possible.  This environment includes widely accessible source code
> repositories, published artifacts, ticket tracking, and extensive
> documentation. We also encourage contributions and frequent debate and
> hold regular, collaborative discussions through e-mail, chat rooms,
> and in-person meet-ups.  We are committed to the ideals of open source
> software and will eagerly seek out mentors and sponsors who can help
> us quickly come up to speed.
>
> === Homogenous Developers ===
> The initial committers of NiFi come from a limited set of entities
> though we are committed to recruiting and developing additional
> committers from a broad spectrum of industries and backgrounds.
>
> === Reliance on Salaried Developers ===
> We expect NiFi development to continue on salaried time and through
> volunteer time.  The initial committers are paid by their employers to
> contribute to this project.  We are committed to developing and
> recruiting participation from developers both salaried and
> non-salaried.
>
> === Relationship with other Apache Projects ===
> As described in the alignment section, NiFi is already heavily
> dependent on other ASF projects and we anticipate further dependence
> and integration with new and emerging projects in the Apache family.
>
> === An Excessive Fascination with the Apache Brand ===
> We respect the laudable Apache brand and that is certainly a factor in
> the decision to propose NiFi for the Apache Incubator.  We intend to
> provide a great deal of energy and capability to the ASF through this
> project.  We will be sensitive to and respectful of any overuse of the
> Apache brand and ensure our focus remains on how we benefit the Apache
> community.
>
> === Documentation ===
> At this time there is no NiFi documentation on the web.  However, we
> have extensive documentation included within the application that
> details usage of the many functions.  We will be rapidly expanding the
> available documentation to cover things like installation, developer
> guide, frequently asked questions, best practices, and more.  This
> documentation will be posted to the NiFi wiki at apache.org.
>
> == Initial Source ==
> NiFi has been in active development since late 2006 with contributions
> from dozens of developers and feedback from hundreds of users and
> developers.  The core codebase is written in Java and includes
> detailed Javadocs and feature documentation.
>
> == Source and Intellectual Property Submission ==
> Previously referred to as Niagarafiles, the NiFi code and
> documentation materials will be submitted by the National Security
> Agency.  NiFi has been developed by a mix of government employees and
> private companies under government contract.  Material developed by
> the government employees is in the public domain and no U.S. copyright
> exists in works of the federal government.  For the contractor
> developed material in the initial submission, the U.S. Government has
> sufficient authority to open source per DFARS 252.227-7014.  NSA has
> submitted the Software Grant Agreement and Corporate Contributor
> License Agreement to the Apache Software Foundation.
>
> == External Dependencies ==
> We have at least one dependency on an LGPL library which we will
> promptly address.  Otherwise, we believe all current dependencies are
> compatible with the ASF guidelines.  Our dependency licenses come from
> the following license styles:  Apache v 2.0, BSD, Public Domain,
> Eclipse Public v1, MIT, CDDL v1.
>
> == Cryptography ==
> Consistent with http://www.apache.org/licenses/exports/ we believe
> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
> cryptographic algorithms but is designed to use algorithms provided by
> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
> These cryptographic algorithm providers are used to support SSL,
> SSH/SFTP, and the encryption and decryption of sensitive properties.
> In the event that it becomes necessary we will engage with appropriate
> Apache members to ensure we file any necessary paperwork or clarified
> any cryptographic export license concerns.
>
> == Required Resources ==
> === Mailing Lists ===
>    * user@nifi.incubator.apache.org
>    * dev@nifi.incubator.apache.org
>    * private@nifi.incubator.apache.org
>    * commits@nifi.incubator.apache.org
>
> === Source Control ===
> NiFi requests use of Git for source control
> (git://git.apache.org/nifi.git).  We request a writeable Git repo for
> NiFi with mirroring to be setup to Github through INFRA.  We request
> sponsor Benson Margulies (bimargulies) to assist with creating the
> INFRA ticket for this.
>
> === Issue Tracking ===
> JIRA NiFi (NIFI)
>
> === Initial Committers ===
>    * Brandon Devries<brandon.devries at gmail dot com>
>    * Matt Gilman<matt.c.gilman at gmail dot com>
>    * Tony Kurc<trkurc at gmail dot com>
>    * Mark Payne<markap14 at hotmail dot com>
>    * Adam Taft<adam at adamtaft dot com>
>    * Joseph Witt<joewitt at gmail dot com>
>
> === Affiliations ===
>    * Brandon Devries (Requitest, Inc.)
>    * Matt Gilman (Raytheon)
>    * Tony Kurc (National Security Agency)
>    * Mark Payne (Sotera Defense Solutions, Inc.)
>    * Adam Taft (Requitest, Inc.)
>    * Joseph Witt (National Security Agency)
>
> == Sponsors ==
> === Champion ===
>    * Benson Margulies (Basis Technology)<bimargulies at apache dot org>
>
> === Nominated Mentors ===
>    * Drew Farris (Booz Allen Hamilton)<drew at apache dot org>
>    * Brock Noland (Cloudera)<brock at apache dot org>
>    * Billie Rinaldi (Hortonworks)<billie at apache dot org>
>
> === Sponsoring Entity ===
> We request the Apache Incubator to sponsor this project.
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [PROPOSAL] NiFi for Incubation

[Jim Jagielski <ji...@jaguNET.com>]

very, VERY cool!

> On Nov 19, 2014, at 9:02 PM, Joe Witt <jo...@gmail.com> wrote:
> 
> Hello,
> 
> 
> I would like to propose NiFi as an Apache Incubator Project.
> 
> In addition to the copy provided below the Wiki version of the
> proposal can be found here:
> http://wiki.apache.org/incubator/NiFiProposal
> 
> Thanks
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [PROPOSAL] NiFi for Incubation

[Michael Ridley <mr...@clouderagovt.com>]

Very excited to see this proposal! 

Michael Ridley

Sent from my mobile.
Pardon any spelling errors.

> On Nov 19, 2014, at 8:02 PM, Joe Witt <jo...@gmail.com> wrote:
> 
> Hello,
> 
> 
> I would like to propose NiFi as an Apache Incubator Project.
> 
> In addition to the copy provided below the Wiki version of the
> proposal can be found here:
> http://wiki.apache.org/incubator/NiFiProposal
> 
> Thanks
> 
> Joe
> 
> 
> = NiFi Proposal =
> 
> == Abstract ==
> NiFi is a dataflow system based on the concepts of flow-based programming.
> 
> == Proposal ==
> NiFi supports powerful and scalable directed graphs of data routing,
> transformation, and system mediation logic.  Some of the high-level
> capabilities and objectives of NiFi include:
>  * Web-based user interface for seamless experience between design,
> control, feedback, and monitoring of data flows
>  * Highly configurable along several dimensions of quality of service
> such as loss tolerant versus guaranteed delivery, low latency versus
> high throughput, and priority based queuing
>  * Fine-grained data provenance for all data received, forked,
> joined, cloned, modified, sent, and ultimately dropped as data reaches
> its configured end-state
>  * Component-based extension model along well defined interfaces
> enabling rapid development and effective testing
> 
> == Background ==
> Reliable and effective dataflow between systems can be difficult
> whether you're running scripts on a laptop or have a massive
> distributed computing system operated by numerous teams and
> organizations.  As the volume and rate of data grows and as the number
> of systems, protocols, and formats increase and evolve so too does the
> complexity and need for greater insight and agility.  These are the
> dataflow challenges that NiFi was built to tackle.
> 
> NiFi is designed in a manner consistent with the core concepts
> described in flow-based programming as originally documented by J.
> Paul Morrison in the 1970s.  This model lends itself well to visual
> diagramming, concurrency, componentization, testing, and reuse.  In
> addition to staying close to the fundamentals of flow-based
> programming, NiFi provides integration system specific features such
> as: guaranteed delivery; back pressure; ability to gracefully handle
> backlogs and data surges; and an operator interface that enables
> on-the-fly data flow generation, modification, and observation.
> 
> == Rationale ==
> NiFi provides a reliable, scalable, manageable and accountable
> platform for developers and technical staff to create and evolve
> powerful data flows.  Such a system is useful in many contexts
> including large-scale enterprise integration, interaction with cloud
> services and frameworks, business to business, intra-departmental, and
> inter-departmental flows.  NiFi fits well within the Apache Software
> Foundation (ASF) family as it depends on numerous ASF projects and
> integrates with several others.  We also anticipate developing
> extensions for several other ASF projects such as Cassandra, Kafka,
> and Storm in the near future.
> 
> == Initial Goals ==
>  * Ensure all dependencies are compliant with Apache License version
> 2.0 and all that all code and documentation artifacts have the correct
> Apache licensing markings and notice.
>  * Establish a formal release process and schedule, allowing for
> dependable release cycles in a manner consistent with the Apache
> development process.
>  * Determine and establish a mechanism, possibly including a
> sub-project construct, that allows for extensions to the core
> application to occur at a pace that differs from the core application
> itself.
> 
> == Current Status ==
> === Meritocracy ===
> An integration platform is only as good as its ability to integrate
> systems in a reliable, timely, and repeatable manner.  The same can be
> said of its ability to attract talent and a variety of perspectives as
> integration systems by their nature are always evolving.  We will
> actively seek help and encourage promotion of influence in the project
> through meritocracy.
> 
> === Community ===
> Over the past several years, NiFi has developed a strong community of
> both developers and operators within the U.S. government.  We look
> forward to helping grow this to a broader base of industries.
> 
> === Core Developers ===
> The initial core developers are employed by the National Security
> Agency and defense contractors.  We will work to grow the community
> among a more diverse set of developers and industries.
> 
> === Alignment ===
> From its inception, NiFi was developed with an open source philosophy
> in mind and with the hopes of eventually being truly open sourced.
> The Apache way is consistent with the approach we have taken to date.
> The ASF clearly provides a mature and effective environment for
> successful development as is evident across the spectrum of well-known
> projects.  Further, NiFi depends on numerous ASF libraries and
> projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop,
> HttpClient, Jakarta and Maven.  We also anticipate extensions and
> dependencies with several more ASF projects, including Accumulo, Avro,
> Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others.
> This existing alignment with Apache and the desired community makes
> the Apache Incubator a good fit for NiFi.
> 
> == Known Risks ==
> === Orphaned Products ===
> Risk of orphaning is minimal.  The project user and developer base is
> substantial, growing, and there is already extensive operational use
> of NiFi.
> 
> === Inexperience with Open Source ===
> The initial committers to NiFi have limited experience with true open
> source software development.  However, despite the project origins
> being from closed source development we have modelled our behavior and
> community development on The Apache Way to the greatest extent
> possible.  This environment includes widely accessible source code
> repositories, published artifacts, ticket tracking, and extensive
> documentation. We also encourage contributions and frequent debate and
> hold regular, collaborative discussions through e-mail, chat rooms,
> and in-person meet-ups.  We are committed to the ideals of open source
> software and will eagerly seek out mentors and sponsors who can help
> us quickly come up to speed.
> 
> === Homogenous Developers ===
> The initial committers of NiFi come from a limited set of entities
> though we are committed to recruiting and developing additional
> committers from a broad spectrum of industries and backgrounds.
> 
> === Reliance on Salaried Developers ===
> We expect NiFi development to continue on salaried time and through
> volunteer time.  The initial committers are paid by their employers to
> contribute to this project.  We are committed to developing and
> recruiting participation from developers both salaried and
> non-salaried.
> 
> === Relationship with other Apache Projects ===
> As described in the alignment section, NiFi is already heavily
> dependent on other ASF projects and we anticipate further dependence
> and integration with new and emerging projects in the Apache family.
> 
> === An Excessive Fascination with the Apache Brand ===
> We respect the laudable Apache brand and that is certainly a factor in
> the decision to propose NiFi for the Apache Incubator.  We intend to
> provide a great deal of energy and capability to the ASF through this
> project.  We will be sensitive to and respectful of any overuse of the
> Apache brand and ensure our focus remains on how we benefit the Apache
> community.
> 
> === Documentation ===
> At this time there is no NiFi documentation on the web.  However, we
> have extensive documentation included within the application that
> details usage of the many functions.  We will be rapidly expanding the
> available documentation to cover things like installation, developer
> guide, frequently asked questions, best practices, and more.  This
> documentation will be posted to the NiFi wiki at apache.org.
> 
> == Initial Source ==
> NiFi has been in active development since late 2006 with contributions
> from dozens of developers and feedback from hundreds of users and
> developers.  The core codebase is written in Java and includes
> detailed Javadocs and feature documentation.
> 
> == Source and Intellectual Property Submission ==
> Previously referred to as Niagarafiles, the NiFi code and
> documentation materials will be submitted by the National Security
> Agency.  NiFi has been developed by a mix of government employees and
> private companies under government contract.  Material developed by
> the government employees is in the public domain and no U.S. copyright
> exists in works of the federal government.  For the contractor
> developed material in the initial submission, the U.S. Government has
> sufficient authority to open source per DFARS 252.227-7014.  NSA has
> submitted the Software Grant Agreement and Corporate Contributor
> License Agreement to the Apache Software Foundation.
> 
> == External Dependencies ==
> We have at least one dependency on an LGPL library which we will
> promptly address.  Otherwise, we believe all current dependencies are
> compatible with the ASF guidelines.  Our dependency licenses come from
> the following license styles:  Apache v 2.0, BSD, Public Domain,
> Eclipse Public v1, MIT, CDDL v1.
> 
> == Cryptography ==
> Consistent with http://www.apache.org/licenses/exports/ we believe
> NiFi is classified as ECCN 5D002.  NiFi doesn't implement any
> cryptographic algorithms but is designed to use algorithms provided by
> Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc.
> These cryptographic algorithm providers are used to support SSL,
> SSH/SFTP, and the encryption and decryption of sensitive properties.
> In the event that it becomes necessary we will engage with appropriate
> Apache members to ensure we file any necessary paperwork or clarified
> any cryptographic export license concerns.
> 
> == Required Resources ==
> === Mailing Lists ===
>  * user@nifi.incubator.apache.org
>  * dev@nifi.incubator.apache.org
>  * private@nifi.incubator.apache.org
>  * commits@nifi.incubator.apache.org
> 
> === Source Control ===
> NiFi requests use of Git for source control
> (git://git.apache.org/nifi.git).  We request a writeable Git repo for
> NiFi with mirroring to be setup to Github through INFRA.  We request
> sponsor Benson Margulies (bimargulies) to assist with creating the
> INFRA ticket for this.
> 
> === Issue Tracking ===
> JIRA NiFi (NIFI)
> 
> === Initial Committers ===
>  * Brandon Devries <brandon.devries at gmail dot com>
>  * Matt Gilman <matt.c.gilman at gmail dot com>
>  * Tony Kurc <trkurc at gmail dot com>
>  * Mark Payne <markap14 at hotmail dot com>
>  * Adam Taft <adam at adamtaft dot com>
>  * Joseph Witt <joewitt at gmail dot com>
> 
> === Affiliations ===
>  * Brandon Devries (Requitest, Inc.)
>  * Matt Gilman (Raytheon)
>  * Tony Kurc (National Security Agency)
>  * Mark Payne (Sotera Defense Solutions, Inc.)
>  * Adam Taft (Requitest, Inc.)
>  * Joseph Witt (National Security Agency)
> 
> == Sponsors ==
> === Champion ===
>  * Benson Margulies (Basis Technology) <bimargulies at apache dot org>
> 
> === Nominated Mentors ===
>  * Drew Farris (Booz Allen Hamilton) <drew at apache dot org>
>  * Brock Noland (Cloudera) <brock at apache dot org>
>  * Billie Rinaldi (Hortonworks) <billie at apache dot org>
> 
> === Sponsoring Entity ===
> We request the Apache Incubator to sponsor this project.

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org