Re: [PROPOSAL] Fluo

[Christopher <ct...@apache.org>]

On Tue, Apr 19, 2016 at 1:21 PM Christopher <ct...@apache.org> wrote:

> For the incubator consideration, the Fluo developers submit the following
> proposal:
> https://wiki.apache.org/incubator/FluoProposal
>
> = Fluo Proposal =
>
> == Abstract ==
>
> Fluo is a distributed system for incrementally processing large data sets
> stored in Accumulo.
>
> == Proposal ==
>
> Fluo is a distributed transaction and notification system that enables the
> incremental processing of large data sets. Its transaction system allows
> for concurrent, cross-node updates to data stored in Accumulo. Its
> notification system enables developers to write code to be executed when
> observed data changes. Fluo provides a core API to perform transactional
> updates using minimalistic get/set methods. Fluo also provides a higher
> order recipes API that builds on the core API to support more complex
> methods for transactional updates.
>
> == Background ==
>
> Several frameworks exist for batch (i.e Spark, MapReduce) and stream (i.e
> Storm, Spark Streaming) processing of data. While batch and stream
> processing have strong use cases, they are not suited for joining incoming
> data in real-time to a large existing data set. To fill this need, Google
> developed an incremental processing system called Percolator and described
> it in the paper, ''Large-scale Incremental Processing Using Distributed
> Transactions and Notifications''<<FootNote(USENIX (2010),
> http://research.google.com/pubs/pub36726.html)>>.
>
> == Rationale ==
>
> Fluo fills the need for cross-row (and cross-node) transactions in
> Accumulo by providing it with an open source implementation of Percolator.
> Fluo also satisfies a gap in Accumulo’s ability to incrementally process
> data. Fluo also provides a novel recipes API which offers higher level
> abstractions for transactional updates.
>
> == Current Status ==
>
> Fluo currently exists as an open source project on GitHub and has been in
> active development since 2013. The project has made an alpha release and
> two beta releases. The major features of Fluo outlined in this proposal
> have been implemented. Several example Fluo applications have been created
> and run successfully on clusters (up to 24 nodes).
>
> === Meritocracy ===
>
> The Fluo project operates as a meritocracy and will continue to do so
> because we feel that a project comprised of a diverse set of committers
> will thrive. Therefore, we welcome new contributors and encourage them on
> their path to committership.
>
> === Community ===
>
> Fluo is currently being used by a subset of the Accumulo community. The
> initial developers have been responsive to external contributions through
> pull requests and issues on GitHub. As Fluo releases a stable 1.0 version
> that is production-ready, we expect this community to grow. To encourage
> growth, we have created a project website with documentation, given talks
> at Meetups and the Accumulo Summit, and engaged with new users on GitHub
> and the Fluo mailing list.
>
> === Core Developers ===
>
> The project was started by Keith Turner (an Apache Member and
> committer/PMC on Gora and Accumulo) in 2013, and the development has
> primarily consisted of his and Mike Walch’s continued efforts. Additional
> developers have contributed over time, which has led to new committers.
>
> === Alignment ===
>
> Fluo is closely linked to the Accumulo community, and fits well within the
> larger Hadoop ecosystem at Apache. Fluo utilizes several Apache projects,
> such as Accumulo, YARN, Twill, and ZooKeeper. Enabling closer collaboration
> between these communities through its coexistence within the ASF would help
> further drive the success of them all.
>
> In addition to our technical ties to other ASF projects, our development
> philosophy aligns with Apache philosophies. Based on our experience with
> existing Apache projects, we are interested in establishing formal
> governance with a PMC and community bylaws, which we feel would best be
> done within Apache.
>
> == Known Risks ==
>
> === Orphaned Products ===
>
> Fluo could be orphaned if the project fails to gain adoption and the core
> developers abandon their interest (this is not anticipated). This risk can
> be mitigated by attracting more committers and developing further
> documentation to ease adoption.
>
> === Inexperience with Open Source ===
>
> Fluo has been an open source project on GitHub from the start of its
> development. Several Fluo developers are committers on other ASF projects
> as well as open source projects outside ASF, and understand open source
> development.
>
> === Homogeneous Developers ===
>
> The initial committers work for different employers. We hope add more
> developers from other employers and industries.
>
> === Reliance on Salaried Developers ===
>
> While most of the initial committers are paid to work on Fluo, there have
> been many contributions from developers working independently.
>
> === Relationships with Other Apache Products ===
>
> Fluo uses Accumulo, Hadoop (HDFS & YARN), Twill, ZooKeeper, Curator,
> Thrift, and various Commons libraries. During development, contributions
> have been made to some of these Apache projects to better support Fluo use
> cases.
>
> === Apache Brand ===
>
> While we recognize the impact of the Apache brand, we feel that Fluo would
> fit well in Apache because of its relationship to other Apache projects and
> because we share the ASF values of meritocracy and community over code.
>
> == Documentation ==
>
> Information about Fluo can be found on the project website at
> http://fluo.io/. This includes:
>
>  * General documentation - http://fluo.io/docs/
>  * API documentation - http://fluo.io/apidocs/
>  * Release notes - http://fluo.io/release-notes/
>  * Blog posts - http://fluo.io/blog/
>
> == Initial Source ==
>
> The initial source code is publicly available as an open source project on
> GitHub at https://github.com/fluo-io/fluo
>
> Supplemental repositories also exist on GitHub at
> https://github.com/fluo-io and some of those will become part of the
> initial code base (perhaps in separate repositories).
>
> == Source and Intellectual Property Submission Plan ==
>
> All of the Fluo’s source code is available under the Apache License,
> Version 2.
>
> The Fluo logo was designed and contributed to the Fluo project, for use by
> the project, and the contributors would like it to remain the logo of the
> project within the ASF, granting any necessary rights to the ASF, while
> continuing to use the logo on Fluo-related historical sites and project
> pages (such as Fluo’s current GitHub site).
>
> == External Dependencies ==
>
> Fluo has made it a point from its beginning to use dependencies which are
> compatible with the expectations of an ASF project. The following are its
> current dependencies, grouped by license.
>
> Apache License, Version 2.0
>  * accumulo
>  * commons-{collections,configuration,io}
>  * curator
>  * dropwizard metrics
>  * easymock
>  * guava
>  * hadoop
>  * jcommander
>  * maven
>  * thrift
>  * twill
>  * zookeeper
>
> BSD License (2-Clause)
>  * HdrHistogram
>
> Eclipse Public License - v 1.0
>  * junit (not bundled)
>  * logback (binary bundling only)
>
> MIT License (Expat)
>  * slf4j
>
> == Cryptography ==
>
> none
>
> == Required Resources ==
>
> === Mailing Lists ===
>
>    * private at fluo.incubator.apache.org
>    * dev at fluo.incubator.apache.org
>    * notifications at fluo.incubator.apache.org
>
> === Git Repository ===
>
>    * https://git-wip-us.apache.org/repos/asf/incubator-fluo.git
>      (The developers will use a git-based site for project documentation
> in the ''asf-site'' branch of the repo.)
>    * https://git-wip-us.apache.org/repos/asf/incubator-fluo-recipes.git
>
> === Issue Tracking ===
>
>    * https://issues.apache.org/jira/browse/FLUO
>      (Currently, the developers rely on GitHub issues. If possible, GitHub
> integration for issue tracking would be preferred. If this is possible, the
> Fluo developers could work with INFRA to transfer the existing GitHub
> repositories to the Apache GitHub organization to bring the existing GitHub
> issues.)
>
> === Continuous Integration ===
>
>    * Travis CI on the GitHub mirror is fine (flag set to build only if
> ''.travis.yml'' file is present)
>
> == Initial Committers ==
>
>  * Keith Turner (kturner at apache dot org)
>  * Mike Walch (mike.walch at ptech-llc dot com)
>  * Corey Nolet (cjnolet at apache dot org)
>  * Christopher Tubbs (ctubbsii at apache dot org)
>  * Josh Elser (elserj at apache dot org)
>
> == Affiliations ==
>
>  * Keith Turner (Peterson Technologies, ASF Member, Accumulo PMC, Gora PMC)
>  * Mike Walch (Peterson Technologies)
>  * Corey Nolet (Tetra Concepts LLC, Accumulo PMC)
>  * Christopher Tubbs (U.S. Government, ASF Member, Accumulo PMC)
>  * Josh Elser (Hortonworks, ASF Member, Accumulo PMC, Calcite PMC, IPMC)
>
> == Sponsors ==
>
> === Champion ===
>
>  * Billie Rinaldi (billie at apache dot org) has volunteered to be our
> Champion
>
> === Nominated Mentors ===
>
>  * Billie Rinaldi (billie at apache dot org)
>  * Benson Margulies (bimargulies at apache dot org)
>  * Lewis John McGibbney (lewiscmc at apache dot org)
>  * Chris Mattmann (mattmann at apache dot org)
>
> === Sponsoring Entity ===
>
>  * The Fluo team requests sponsorship from the Incubator PMC
>
>
Minor update:

We just updated the mentors section of the proposal to include those
persons with whom we've already spoken about mentoring us, and to exclude
those with whom we didn't get a chance to personally ask.

The new list includes {drew, billie, elserj}@apache.org

Re: [PROPOSAL] Fluo

[Henry Saputra <he...@gmail.com>]

The proposal looks good.

Bit small in mentor list but as long as the mentors are actively engage
should work great!

- Henry

On Tue, May 10, 2016 at 3:58 PM, Christopher <ct...@apache.org> wrote:

> On Tue, May 10, 2016 at 3:52 PM Christopher <ct...@apache.org> wrote:
>
> > On Tue, Apr 19, 2016 at 1:21 PM Christopher <ct...@apache.org> wrote:
> >
> >> For the incubator consideration, the Fluo developers submit the
> following
> >> proposal:
> >> https://wiki.apache.org/incubator/FluoProposal
> >>
> >> [snip]
>
> >
> > Minor update:
> >
> > We just updated the mentors section of the proposal to include those
> > persons with whom we've already spoken about mentoring us, and to exclude
> > those with whom we didn't get a chance to personally ask.
> >
> > The new list includes {drew, billie, elserj}@apache.org
> >
> >
> To clarify, we changed up the mentors listed on the proposal because of a
> misunderstanding on our part about who to list. Those persons currently
> listed are those we actually discussed mentoring with us, and who expressed
> a willingness to do so.
>
> If there's any other persons who'd be interested in offering any mentoring
> for Fluo, we'd love to have you as well, if our proposal is accepted into
> the incubator.
>

Re: [PROPOSAL] Fluo

[Christopher <ct...@apache.org>]

On Tue, May 10, 2016 at 3:52 PM Christopher <ct...@apache.org> wrote:

> On Tue, Apr 19, 2016 at 1:21 PM Christopher <ct...@apache.org> wrote:
>
>> For the incubator consideration, the Fluo developers submit the following
>> proposal:
>> https://wiki.apache.org/incubator/FluoProposal
>>
>> [snip]

>
> Minor update:
>
> We just updated the mentors section of the proposal to include those
> persons with whom we've already spoken about mentoring us, and to exclude
> those with whom we didn't get a chance to personally ask.
>
> The new list includes {drew, billie, elserj}@apache.org
>
>
To clarify, we changed up the mentors listed on the proposal because of a
misunderstanding on our part about who to list. Those persons currently
listed are those we actually discussed mentoring with us, and who expressed
a willingness to do so.

If there's any other persons who'd be interested in offering any mentoring
for Fluo, we'd love to have you as well, if our proposal is accepted into
the incubator.