You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@incubator.apache.org by Jitendra Pandey <ji...@hortonworks.com> on 2016/11/23 08:29:28 UTC
[DISCUSS] Concur for Apache Incubator
Dear all,
We would like to start a discussion on accepting Concur, a java implementation for RAFT consensus protocol, into the Apache Incubator.
Here is the link to the proposal on the wiki
https://wiki.apache.org/incubator/ConcurProposal
The text of the proposal is included below.
Thanks,
Jitendra
--
= Concur Proposal =
== Abstract ==
Concur is an open source java implementation for RAFT consensus protocol[1]. RAFT is being used successfully as an alternative to Paxos to implement a consistently replicated log. RAFT is proven to be safe and is designed to be simpler to understand.
== Proposal ==
Concur is implemented as an independent RAFT library that can be used by any application to manage their replicated logs or replicated state machines. The implementation closely follows the original design proposed in the RAFT paper.
== Background ==
Replicated log is a frequently used technique in distributed systems for reliability and parallelism. The consistency of replicas is an important requirement to ensure the correctness of the applications accessing the data in these replicas. Many algorithms have been proposed and used to maintain consistency and correctness. RAFT paper was proposed in 2012 and has gained significant popularity since then to manage the replicated logs and replicated state in many popular projects. Data replication is a requirement for many projects in ASF ecosystem for example Apache Hadoop, Apache HBase etc. We believe the Concur project can provide an implementation of RAFT that can fulfill the needs similar to these systems and strengthen the ASF ecosystem.
== Rationale ==
There are a few RAFT implementations[2], but none are a part of ASF. Concur is significantly different because one of the critical goals of the project is to use it as a library with pluggability for different RPC, Raft log and state machine implementations. Another important goal of Apache Concur is to provide high throughput for large ingest rates of data, with data pipeline support. This will simplify the problem of data replication across the board. Apache Kudu uses RAFT protocol, but it has its own C++ implementation. Apache DistributedLog project[3] (in incubation) provides a replicated log service. However, Apache Concur is different as it provides a java library that other projects can use to implement their own replicated state machine, without deploying another service.
== Current Status ==
The source code is available with Apache v2 license at https://github.com/hortonworks/concur. Significant amount of code has been added and basic functionality of RAFT is available for testing. The code is still in pre-alpha stage and is currently being tried out in POC mode.
== Meritocracy ==
We plan to invest in supporting meritocracy. We intend to invite additional developers to participate. We will encourage and monitor community participation so that privileges can be extended to those that contribute.
== Community ==
The developers on the initial committers list are experienced in the ASF ecosystem:
* Jing Zhao (jing9 at apache dot org)
* Tsz Wo Nicholas Sze (szetetszwo at apache dot org)
* Li Lu (llu at hortonworks dot com)
* Enis Soztutar (enis at apache dot org)
* Anu Engineer (anu at apache dot org)
* Arpit Agarwal (arpitagarwal at apache dot org)
* Chris Nauroth (cnauroth at apache dot org)
* Jakob Homan (jghoman at apache dot org)
* Mayank Bansal (mayank_bansal at apache dot org)
* Xiaoyu Yao (xyao at apache dot org)
* Chen Liang (cliang at hortonworks dot com)
* Hanisha Koneru (hkoneru at hortonworks dot com)
* Xiaobing Zhou (xiaobingo at gmail dot com)
* Mingliang Liu (liuml07 at apache dot org)
* Uma Maheswara Rao G (umamahesh at apache dot org)
* Jitendra Pandey (jitendra at apache dot org)
== Affiliations ==
* Jing Zhao, Tsz Wo Nicholas Sze, Li Lu, Enis Soztutar, Anu Engineer, Arpit Agarwal, Xiaoyu Yao, Chen Liang, Hanisha Koneru, Mingliang Liu, Jitendra Pandey, Devaraj Das
* Hortonworks
* Uma Gangumalla, Intel
* Mayank Bansal, Uber
* Jakob Homan, OfferUp
* Chris Nauroth, Disney
== Alignment ==
We believe that Concur will address the requirements of several projects and communities in the Apache ecosystem and will gain great adoption.
== Known Risks ==
=== Orphaned Products ===
The contributors are leading users and vendors in the Apache Hadoop ecosystem, with significant open source experience, so the risk of being orphaned is relatively low. The project could be at risk if vendors decided to change their strategies in the market. In such an event, the current committers plan to continue working on the project on their own time, though the progress will likely be slower. We plan to mitigate this risk by encouraging and recruiting additional committers. Since replicated log is a very common technique useful for large number of applications, we believe many developers would like to join and contribute to the project.
=== Inexperience with Open Source ===
The initial committers include veteran Apache Members (Committers, PMC Members and Apache Members) and other developers who have varying degrees of experience with open source projects. All have been involved with source code that has been released under an open source license, and also have experience developing code with an open source development process.
=== Homogenous Developers ===
The initial list of committers include senior Apache developers from multiple different organizations including Hortonworks, OfferUp, Intel, Uber etc. We believe this project will benefit many different projects and therefore more and more diverse set of developers will join over time.
=== Reliance on Salaried Developers ===
It is expected that Concur development will occur on both salaried time and on volunteer time, after hours. The majority of initial committers are paid by their employer to contribute to this project. However, they are all passionate about the project, and we are confident that the project will continue even if no salaried developers contribute to the project. We are committed to recruiting additional committers including non-salaried developers.
=== Relationships with Other Apache Products ===
Most of the initial developers are active participants in Apache Hadoop community and we expect adoption of Concur by Hadoop Community. However, Concur is a generic RAFT library and we will strive to take it to more general adoption. This project depends on Apache Maven, Apache Commons {collections,configuration,io} and Apache Hadoop.
== Initial Source ==
https://github.com/hortonworks/concur
== External Dependencies ==
* Apache Maven
* Apache Commons {collections,configuration,io}
* Apache Hadoop
* Netty (Apache 2.0)
* Guava (Apache 2.0)
* Protobuf (Google 2014)
* gRPC (Google 2015)
* slf4j (MIT License)
* Mockito (MIT)
* junit (EPL 1.0)
* FindBugs (LGPL)
== Required Resources ==
=== Mailing List ===
* concur-private
* concur-dev
* concur-user
* concur-issues
=== Git Repository ===
Git is the preferred source control system: git://git.apache.org/concur
=== Issue Tracking ===
JIRA Concur (Concur)
== Sponsors ==
=== Champion ===
Jitendra Pandey (jitendra)
=== Nominated Mentors ===
* Chris Nauroth (cnauroth)
* Devaraj Das (devaraj)
* Jakob Homan (jghoman)
* Uma Maheswara Rao G (umamahesh)
=== Sponsoring Entity ===
Incubator PMC
== Reference ==
* [1] Diego Ongaro and John Ousterhout. 2014. In search of an understandable consensus algorithm. In Proceedings of the 2014 USENIX conference on USENIX Annual Technical Conference (USENIX ATC'14), Garth Gibson and Nickolai Zeldovich (Eds.). USENIX Association, Berkeley, CA, USA, 305-320.
* [2] https://raft.github.io/
* [3] Apache DistributedLog (incubating) http://distributedlog.incubator.apache.org/
Re: [DISCUSS] Concur for Apache Incubator
Posted by Willem Jiang <wi...@gmail.com>.
Yeah, the project still looks like an internal project, I hardly can find
any document about it.
I just went through the github project, there is only just one README. I
cannot find any wiki pages or other document talking about the roadmap or
high level design. I guess the Raft site[1] could help us understand the
Raft Algorithm, but other user or developer still need some information to
dive in.
[1]https://raft.github.io/
Willem Jiang
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On Wed, Nov 23, 2016 at 10:35 PM, Stian Soiland-Reyes <st...@apache.org>
wrote:
> I think we should in principle aim for also a larger diversity in the
> initial committers for a straight-to-TLP,
> https://github.com/hortonworks/concur/graphs/contributors does not
> show anyone that is not working for Hortonworks - but the proposed
> committers (aka proposed PMC?) have broad ASF experience.
>
> Perhaps the stay in the incubator would be short before graduation,
> after the project shows non-Hortonworks folks are engaged.
>
> Has this proposal or code base been discussed in any of the Hadoop
> lists? I could not find anything in the list archives, but then I'm
> not too familiar with Hadoop. Or perhaps it was discussed elsewhere
> since development started in April?
>
>
> On 23 November 2016 at 14:19, Greg Stein <gs...@gmail.com> wrote:
> > On Wed, Nov 23, 2016 at 5:43 AM, John D. Ament <jo...@apache.org>
> > wrote:
> >
> >> On Wed, Nov 23, 2016 at 6:40 AM Roman Shaposhnik <ro...@shaposhnik.org>
> >> wrote:
> >>
> >> > On Wed, Nov 23, 2016 at 11:58 AM, Jean-Baptiste Onofré <
> jb@nanthrax.net>
> >> > wrote:
> >> > > Hi Jitendra,
> >> > >
> >> > > I gonna take a deeper look.
> >> > >
> >> > > My first feedback would be about the name: Concur is
> traveling/expense
> >> > > solution (https://www.concursolutions.com/).
> >> > >
> >> > > To avoid confusion, maybe it would make sense to choose another
> name.
> >> >
> >> > FIWI: these were my thoughts exactly. In fact, for a second there I
> >> thought
> >> > that SAP was donating Concur codebase.
> >> >
> >>
> >> Good to know I wasn't the only one.
> >>
> >> Jitendra, So considering how strong of a community this is already and
> the
> >> success of recent TLP direct projects, what goals do you have for
> >> incubation? And why incubation over straight to TLP?
> >
> >
> > It is an external codebase, which sets a high bar. The initial committers
> > do seem to mostly have ICLAs (haven't looked at all). The Board would
> > likely want to see a large majority of them also being ASF Members,
> before
> > considering straight-to-TLP.
> >
> > Cheers,
> > -g
>
>
>
> --
> Stian Soiland-Reyes
> http://orcid.org/0000-0001-9842-9718
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: [DISCUSS] Concur for Apache Incubator
Posted by Stian Soiland-Reyes <st...@apache.org>.
I think we should in principle aim for also a larger diversity in the
initial committers for a straight-to-TLP,
https://github.com/hortonworks/concur/graphs/contributors does not
show anyone that is not working for Hortonworks - but the proposed
committers (aka proposed PMC?) have broad ASF experience.
Perhaps the stay in the incubator would be short before graduation,
after the project shows non-Hortonworks folks are engaged.
Has this proposal or code base been discussed in any of the Hadoop
lists? I could not find anything in the list archives, but then I'm
not too familiar with Hadoop. Or perhaps it was discussed elsewhere
since development started in April?
On 23 November 2016 at 14:19, Greg Stein <gs...@gmail.com> wrote:
> On Wed, Nov 23, 2016 at 5:43 AM, John D. Ament <jo...@apache.org>
> wrote:
>
>> On Wed, Nov 23, 2016 at 6:40 AM Roman Shaposhnik <ro...@shaposhnik.org>
>> wrote:
>>
>> > On Wed, Nov 23, 2016 at 11:58 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>> > wrote:
>> > > Hi Jitendra,
>> > >
>> > > I gonna take a deeper look.
>> > >
>> > > My first feedback would be about the name: Concur is traveling/expense
>> > > solution (https://www.concursolutions.com/).
>> > >
>> > > To avoid confusion, maybe it would make sense to choose another name.
>> >
>> > FIWI: these were my thoughts exactly. In fact, for a second there I
>> thought
>> > that SAP was donating Concur codebase.
>> >
>>
>> Good to know I wasn't the only one.
>>
>> Jitendra, So considering how strong of a community this is already and the
>> success of recent TLP direct projects, what goals do you have for
>> incubation? And why incubation over straight to TLP?
>
>
> It is an external codebase, which sets a high bar. The initial committers
> do seem to mostly have ICLAs (haven't looked at all). The Board would
> likely want to see a large majority of them also being ASF Members, before
> considering straight-to-TLP.
>
> Cheers,
> -g
--
Stian Soiland-Reyes
http://orcid.org/0000-0001-9842-9718
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Concur for Apache Incubator
Posted by Greg Stein <gs...@gmail.com>.
On Wed, Nov 23, 2016 at 5:43 AM, John D. Ament <jo...@apache.org>
wrote:
> On Wed, Nov 23, 2016 at 6:40 AM Roman Shaposhnik <ro...@shaposhnik.org>
> wrote:
>
> > On Wed, Nov 23, 2016 at 11:58 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> > > Hi Jitendra,
> > >
> > > I gonna take a deeper look.
> > >
> > > My first feedback would be about the name: Concur is traveling/expense
> > > solution (https://www.concursolutions.com/).
> > >
> > > To avoid confusion, maybe it would make sense to choose another name.
> >
> > FIWI: these were my thoughts exactly. In fact, for a second there I
> thought
> > that SAP was donating Concur codebase.
> >
>
> Good to know I wasn't the only one.
>
> Jitendra, So considering how strong of a community this is already and the
> success of recent TLP direct projects, what goals do you have for
> incubation? And why incubation over straight to TLP?
It is an external codebase, which sets a high bar. The initial committers
do seem to mostly have ICLAs (haven't looked at all). The Board would
likely want to see a large majority of them also being ASF Members, before
considering straight-to-TLP.
Cheers,
-g
Re: [DISCUSS] Concur for Apache Incubator
Posted by "John D. Ament" <jo...@apache.org>.
On Wed, Nov 23, 2016 at 6:40 AM Roman Shaposhnik <ro...@shaposhnik.org>
wrote:
> On Wed, Nov 23, 2016 at 11:58 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> > Hi Jitendra,
> >
> > I gonna take a deeper look.
> >
> > My first feedback would be about the name: Concur is traveling/expense
> > solution (https://www.concursolutions.com/).
> >
> > To avoid confusion, maybe it would make sense to choose another name.
>
> FIWI: these were my thoughts exactly. In fact, for a second there I thought
> that SAP was donating Concur codebase.
>
Good to know I wasn't the only one.
Jitendra, So considering how strong of a community this is already and the
success of recent TLP direct projects, what goals do you have for
incubation? And why incubation over straight to TLP?
>
> In the spirit of bikeshedding I propose Apache Thor (or Heyerdahl) 'cuz
> you know: raft ;-)
>
> Thanks,
> Roman.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: [DISCUSS] Concur for Apache Incubator
Posted by Mark Bretl <as...@gmail.com>.
FYI, the name 'Kontiki' is in use by Kollective Technology, as a former
company name and now brand name [1]
[1] http://www.kontiki.com
--Mark
On Wed, Nov 23, 2016 at 7:55 AM, Alex Harui <ah...@adobe.com> wrote:
>
>
> On 11/23/16, 6:19 AM, "Stian Soiland-Reyes" <st...@apache.org> wrote:
>
> >On 23 November 2016 at 11:40, Roman Shaposhnik <ro...@shaposhnik.org>
> >wrote:
> >> FIWI: these were my thoughts exactly. In fact, for a second there I
> >>thought
> >> that SAP was donating Concur codebase.
> >> In the spirit of bikeshedding I propose Apache Thor (or Heyerdahl) 'cuz
> >> you know: raft ;-)
> >
> >As Norwegian I should not really disagree on that name -- but Thor is
> >already the name of lots of things, including at least two EU projecst
> >https://project-thor.eu/ (Open Research interoperability)
> >http://www.eu-thor.eu/ (Oceanography!)
> >
> >
> >Apache Heyerdahl..?
>
> Thor Heyerdahl wrote a book about a raft trip from South America to the
> South Pacific. His first book is titled "Kon-Tiki". There is already an
> Apache Tika, so maybe Apache Tiki wouldn't work, but maybe Apache ConTiki.
> Not sure if we could use the book title in the project name.
>
> Just painting the bike shed,
> -Alex
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
Re: [DISCUSS] Concur for Apache Incubator
Posted by Alex Harui <ah...@adobe.com>.
On 11/23/16, 6:19 AM, "Stian Soiland-Reyes" <st...@apache.org> wrote:
>On 23 November 2016 at 11:40, Roman Shaposhnik <ro...@shaposhnik.org>
>wrote:
>> FIWI: these were my thoughts exactly. In fact, for a second there I
>>thought
>> that SAP was donating Concur codebase.
>> In the spirit of bikeshedding I propose Apache Thor (or Heyerdahl) 'cuz
>> you know: raft ;-)
>
>As Norwegian I should not really disagree on that name -- but Thor is
>already the name of lots of things, including at least two EU projecst
>https://project-thor.eu/ (Open Research interoperability)
>http://www.eu-thor.eu/ (Oceanography!)
>
>
>Apache Heyerdahl..?
Thor Heyerdahl wrote a book about a raft trip from South America to the
South Pacific. His first book is titled "Kon-Tiki". There is already an
Apache Tika, so maybe Apache Tiki wouldn't work, but maybe Apache ConTiki.
Not sure if we could use the book title in the project name.
Just painting the bike shed,
-Alex
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Concur for Apache Incubator
Posted by Chris Nauroth <cn...@apache.org>.
Responding to a few points in the thread:
> Jitendra, So considering how strong of a community this is already and the
> success of recent TLP direct projects, what goals do you have for
> incubation? And why incubation over straight to TLP?
As hinted by a few others, I would like to see the project's time in the
Incubator focus on establishing a repeatable release process (always tricky
the first few times for a podling), providing sufficient documentation for
usage and contributor onboarding, and establishing community interaction on
the mailing lists. From my knowledge of straight-to-TLP projects, I
believe those spun out of "refactoring" pieces of well-established projects
into new TLPs, so the spin-offs to some degree were able to inherit these
pieces from being previously hosted in the larger project. In this case, I
prefer incubation, because Concur is not a spin-off and therefore still
needs to establish these pieces for itself.
> I think it sounds like an interesting proposal but wonders which
> consumers (and potential developers) it is targeting - I wonder about
> the strong link to Hortonworks and Hadoop -- is Concur relying on
> Hadoop, Hadoop might use Concur, or can Concur be used with many
> things, including Hadoop?
Our goal is to provide a stand-alone library providing a RAFT
implementation. Although Apache Hadoop ecosystem projects are mentioned as
one of the primary targeted use cases, we do not intend to constrain it to
usage only within the Hadoop ecosystem or tightly couple it to any
particular vendor's distribution of the Hadoop ecosystem. I could imagine
it being viable for usage within pieces of Hadoop or in a proprietary
distributed system that I work on at my day job.
It is true that the Concur codebase currently has a Maven dependency on
Hadoop. This was largely a matter of coding convenience to pick up helpful
utilities for configuration management and networking. For those
knowledgeable of Hadoop, I have listed all "org.apache.hadoop" imports
below. You'll see that this is not tightly coupled to deep pieces of HDFS,
YARN, etc., but instead more like helpful utilities. Whether or not the
Hadoop dependency is desirable long-term toward the goal of "stand-alone
library" is a question worthy of consideration for the Concur community,
though I see that as a technical decision point for the community, not a
factor for consideration of whether to accept the incubation proposal.
org.apache.hadoop.classification.InterfaceAudience;
org.apache.hadoop.classification.InterfaceStability;
org.apache.hadoop.conf.Configuration;
org.apache.hadoop.fs.ChecksumException;
org.apache.hadoop.fs.CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_MAX_RETRIES_KEY;
org.apache.hadoop.fs.CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_TIMEOUT_KEY;
org.apache.hadoop.fs.FileUtil;
org.apache.hadoop.io.DataOutputOutputStream;
org.apache.hadoop.io.IOUtils;
org.apache.hadoop.io.MD5Hash;
org.apache.hadoop.io.Writable;
org.apache.hadoop.io.nativeio.NativeIO;
org.apache.hadoop.io.nativeio.NativeIOException;
org.apache.hadoop.io.retry.RetryPolicy;
org.apache.hadoop.ipc.Client.ConnectionId;
org.apache.hadoop.ipc.ProtobufRpcEngineShaded;
org.apache.hadoop.ipc.ProtocolInfo;
org.apache.hadoop.ipc.RPC.RpcInvoker;
org.apache.hadoop.ipc.RPC;
org.apache.hadoop.ipc.RemoteException;
org.apache.hadoop.net.NetUtils;
org.apache.hadoop.security.KerberosInfo;
org.apache.hadoop.security.UserGroupInformation;
org.apache.hadoop.security.token.SecretManager;
org.apache.hadoop.security.token.TokenIdentifier;
org.apache.hadoop.test.GenericTestUtils;
org.apache.hadoop.util.Daemon;
org.apache.hadoop.util.DataChecksum;
org.apache.hadoop.util.ExitUtil;
org.apache.hadoop.util.ProtoUtil;
org.apache.hadoop.util.StringInterner;
org.apache.hadoop.util.StringUtils;
org.apache.hadoop.util.Time;
org.apache.raft.shaded.org.apache.hadoop.ipc.protobuf.ProtobufRpcEngineProtos.RequestHeaderProto;
org.apache.raft.shaded.org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcRequestHeaderProto;
org.apache.raft.shaded.org.apache.hadoop.ipc.protobuf.RpcHeaderProtos.RpcResponseHeaderProto;
--
Chris Nauroth
On Tue, Nov 29, 2016 at 2:59 PM, Jitendra Pandey <ji...@hortonworks.com>
wrote:
> Thanks for the feedback.
> How does Ratis (Latin for raft) sound for the name? A quick search in
> google doesn¹t show it is used. If this is acceptable, I will update the
> proposal.
>
> We are working on documentation for the user api's as well as the design.
>
> We didn¹t consider TLP as the project is very new. We will focus on a
> release cadence and community diversity to qualify for TLP.
>
> Thanks
> jitendra
>
>
> On 11/23/16, 6:19 AM, "Stian Soiland-Reyes" <st...@apache.org> wrote:
>
> >On 23 November 2016 at 11:40, Roman Shaposhnik <ro...@shaposhnik.org>
> >wrote:
> >> FIWI: these were my thoughts exactly. In fact, for a second there I
> >>thought
> >> that SAP was donating Concur codebase.
> >> In the spirit of bikeshedding I propose Apache Thor (or Heyerdahl) 'cuz
> >> you know: raft ;-)
> >
> >As Norwegian I should not really disagree on that name -- but Thor is
> >already the name of lots of things, including at least two EU projecst
> >https://project-thor.eu/ (Open Research interoperability)
> >http://www.eu-thor.eu/ (Oceanography!)
> >
> >
> >Apache Heyerdahl..?
> >
> >http://www.ssb.no/en/befolkning/statistikker/navn says:
> >> There are 410 [Norwegians] with Heyerdahl as their surname
> >
> >(>200 meaning it is not a protected surname in .no :)
> >
> >And 111 businesses - just in Norway - several doing IT stuff:
> >https://w2.brreg.no/enhet/sok/treffliste.jsp?navn=Heyerdahl
> &orgform=0&fylk
> >e=0&kommune=0
> >
> >
> >But we can discuss the proposal even if it has to change its name -
> >that's OK to sort in the very beginning (but is very preferably
> >community-wise to agree before actually moving to the Incubator).
> >
> >
> >I think it sounds like an interesting proposal but wonders which
> >consumers (and potential developers) it is targeting - I wonder about
> >the strong link to Hortonworks and Hadoop -- is Concur relying on
> >Hadoop, Hadoop might use Concur, or can Concur be used with many
> >things, including Hadoop?
> >
> >
> >I see https://github.com/hortonworks/concur is already using
> >org.apache.raft (and before org.apache.hadoop.raft) as a package name
> >- I find this approach for proposals a bit concerning trademark-wise;
> >but then I didn't look closely if this started as a fork/pull request
> >for Hadoop or similar?
> >
> >--
> >Stian Soiland-Reyes
> >http://orcid.org/0000-0001-9842-9718
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >For additional commands, e-mail: general-help@incubator.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
> --
> Chris Nauroth
> <ge...@incubator.apache.org>
>
<ge...@incubator.apache.org>
Re: [DISCUSS] Concur for Apache Incubator
Posted by Jitendra Pandey <ji...@hortonworks.com>.
Thanks for the feedback.
How does Ratis (Latin for raft) sound for the name? A quick search in
google doesn¹t show it is used. If this is acceptable, I will update the
proposal.
We are working on documentation for the user api's as well as the design.
We didn¹t consider TLP as the project is very new. We will focus on a
release cadence and community diversity to qualify for TLP.
Thanks
jitendra
On 11/23/16, 6:19 AM, "Stian Soiland-Reyes" <st...@apache.org> wrote:
>On 23 November 2016 at 11:40, Roman Shaposhnik <ro...@shaposhnik.org>
>wrote:
>> FIWI: these were my thoughts exactly. In fact, for a second there I
>>thought
>> that SAP was donating Concur codebase.
>> In the spirit of bikeshedding I propose Apache Thor (or Heyerdahl) 'cuz
>> you know: raft ;-)
>
>As Norwegian I should not really disagree on that name -- but Thor is
>already the name of lots of things, including at least two EU projecst
>https://project-thor.eu/ (Open Research interoperability)
>http://www.eu-thor.eu/ (Oceanography!)
>
>
>Apache Heyerdahl..?
>
>http://www.ssb.no/en/befolkning/statistikker/navn says:
>> There are 410 [Norwegians] with Heyerdahl as their surname
>
>(>200 meaning it is not a protected surname in .no :)
>
>And 111 businesses - just in Norway - several doing IT stuff:
>https://w2.brreg.no/enhet/sok/treffliste.jsp?navn=Heyerdahl&orgform=0&fylk
>e=0&kommune=0
>
>
>But we can discuss the proposal even if it has to change its name -
>that's OK to sort in the very beginning (but is very preferably
>community-wise to agree before actually moving to the Incubator).
>
>
>I think it sounds like an interesting proposal but wonders which
>consumers (and potential developers) it is targeting - I wonder about
>the strong link to Hortonworks and Hadoop -- is Concur relying on
>Hadoop, Hadoop might use Concur, or can Concur be used with many
>things, including Hadoop?
>
>
>I see https://github.com/hortonworks/concur is already using
>org.apache.raft (and before org.apache.hadoop.raft) as a package name
>- I find this approach for proposals a bit concerning trademark-wise;
>but then I didn't look closely if this started as a fork/pull request
>for Hadoop or similar?
>
>--
>Stian Soiland-Reyes
>http://orcid.org/0000-0001-9842-9718
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>For additional commands, e-mail: general-help@incubator.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Concur for Apache Incubator
Posted by Stian Soiland-Reyes <st...@apache.org>.
On 23 November 2016 at 11:40, Roman Shaposhnik <ro...@shaposhnik.org> wrote:
> FIWI: these were my thoughts exactly. In fact, for a second there I thought
> that SAP was donating Concur codebase.
> In the spirit of bikeshedding I propose Apache Thor (or Heyerdahl) 'cuz
> you know: raft ;-)
As Norwegian I should not really disagree on that name -- but Thor is
already the name of lots of things, including at least two EU projecst
https://project-thor.eu/ (Open Research interoperability)
http://www.eu-thor.eu/ (Oceanography!)
Apache Heyerdahl..?
http://www.ssb.no/en/befolkning/statistikker/navn says:
> There are 410 [Norwegians] with Heyerdahl as their surname
(>200 meaning it is not a protected surname in .no :)
And 111 businesses - just in Norway - several doing IT stuff:
https://w2.brreg.no/enhet/sok/treffliste.jsp?navn=Heyerdahl&orgform=0&fylke=0&kommune=0
But we can discuss the proposal even if it has to change its name -
that's OK to sort in the very beginning (but is very preferably
community-wise to agree before actually moving to the Incubator).
I think it sounds like an interesting proposal but wonders which
consumers (and potential developers) it is targeting - I wonder about
the strong link to Hortonworks and Hadoop -- is Concur relying on
Hadoop, Hadoop might use Concur, or can Concur be used with many
things, including Hadoop?
I see https://github.com/hortonworks/concur is already using
org.apache.raft (and before org.apache.hadoop.raft) as a package name
- I find this approach for proposals a bit concerning trademark-wise;
but then I didn't look closely if this started as a fork/pull request
for Hadoop or similar?
--
Stian Soiland-Reyes
http://orcid.org/0000-0001-9842-9718
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Concur for Apache Incubator
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Wed, Nov 23, 2016 at 11:58 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi Jitendra,
>
> I gonna take a deeper look.
>
> My first feedback would be about the name: Concur is traveling/expense
> solution (https://www.concursolutions.com/).
>
> To avoid confusion, maybe it would make sense to choose another name.
FIWI: these were my thoughts exactly. In fact, for a second there I thought
that SAP was donating Concur codebase.
In the spirit of bikeshedding I propose Apache Thor (or Heyerdahl) 'cuz
you know: raft ;-)
Thanks,
Roman.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] Concur for Apache Incubator
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Jitendra,
I gonna take a deeper look.
My first feedback would be about the name: Concur is traveling/expense
solution (https://www.concursolutions.com/).
To avoid confusion, maybe it would make sense to choose another name.
Regards
JB
On 11/23/2016 09:29 AM, Jitendra Pandey wrote:
> Dear all,
> We would like to start a discussion on accepting Concur, a java implementation for RAFT consensus protocol, into the Apache Incubator.
>
> Here is the link to the proposal on the wiki
> https://wiki.apache.org/incubator/ConcurProposal
>
> The text of the proposal is included below.
>
> Thanks,
> Jitendra
>
> --
>
> = Concur Proposal =
>
> == Abstract ==
> Concur is an open source java implementation for RAFT consensus protocol[1]. RAFT is being used successfully as an alternative to Paxos to implement a consistently replicated log. RAFT is proven to be safe and is designed to be simpler to understand.
>
> == Proposal ==
> Concur is implemented as an independent RAFT library that can be used by any application to manage their replicated logs or replicated state machines. The implementation closely follows the original design proposed in the RAFT paper.
>
> == Background ==
> Replicated log is a frequently used technique in distributed systems for reliability and parallelism. The consistency of replicas is an important requirement to ensure the correctness of the applications accessing the data in these replicas. Many algorithms have been proposed and used to maintain consistency and correctness. RAFT paper was proposed in 2012 and has gained significant popularity since then to manage the replicated logs and replicated state in many popular projects. Data replication is a requirement for many projects in ASF ecosystem for example Apache Hadoop, Apache HBase etc. We believe the Concur project can provide an implementation of RAFT that can fulfill the needs similar to these systems and strengthen the ASF ecosystem.
>
>
> == Rationale ==
> There are a few RAFT implementations[2], but none are a part of ASF. Concur is significantly different because one of the critical goals of the project is to use it as a library with pluggability for different RPC, Raft log and state machine implementations. Another important goal of Apache Concur is to provide high throughput for large ingest rates of data, with data pipeline support. This will simplify the problem of data replication across the board. Apache Kudu uses RAFT protocol, but it has its own C++ implementation. Apache DistributedLog project[3] (in incubation) provides a replicated log service. However, Apache Concur is different as it provides a java library that other projects can use to implement their own replicated state machine, without deploying another service.
>
>
> == Current Status ==
> The source code is available with Apache v2 license at https://github.com/hortonworks/concur. Significant amount of code has been added and basic functionality of RAFT is available for testing. The code is still in pre-alpha stage and is currently being tried out in POC mode.
>
>
> == Meritocracy ==
> We plan to invest in supporting meritocracy. We intend to invite additional developers to participate. We will encourage and monitor community participation so that privileges can be extended to those that contribute.
>
>
> == Community ==
> The developers on the initial committers list are experienced in the ASF ecosystem:
> * Jing Zhao (jing9 at apache dot org)
> * Tsz Wo Nicholas Sze (szetetszwo at apache dot org)
> * Li Lu (llu at hortonworks dot com)
> * Enis Soztutar (enis at apache dot org)
> * Anu Engineer (anu at apache dot org)
> * Arpit Agarwal (arpitagarwal at apache dot org)
> * Chris Nauroth (cnauroth at apache dot org)
> * Jakob Homan (jghoman at apache dot org)
> * Mayank Bansal (mayank_bansal at apache dot org)
> * Xiaoyu Yao (xyao at apache dot org)
> * Chen Liang (cliang at hortonworks dot com)
> * Hanisha Koneru (hkoneru at hortonworks dot com)
> * Xiaobing Zhou (xiaobingo at gmail dot com)
> * Mingliang Liu (liuml07 at apache dot org)
> * Uma Maheswara Rao G (umamahesh at apache dot org)
> * Jitendra Pandey (jitendra at apache dot org)
>
> == Affiliations ==
> * Jing Zhao, Tsz Wo Nicholas Sze, Li Lu, Enis Soztutar, Anu Engineer, Arpit Agarwal, Xiaoyu Yao, Chen Liang, Hanisha Koneru, Mingliang Liu, Jitendra Pandey, Devaraj Das
> * Hortonworks
> * Uma Gangumalla, Intel
> * Mayank Bansal, Uber
> * Jakob Homan, OfferUp
> * Chris Nauroth, Disney
>
> == Alignment ==
> We believe that Concur will address the requirements of several projects and communities in the Apache ecosystem and will gain great adoption.
>
> == Known Risks ==
>
> === Orphaned Products ===
> The contributors are leading users and vendors in the Apache Hadoop ecosystem, with significant open source experience, so the risk of being orphaned is relatively low. The project could be at risk if vendors decided to change their strategies in the market. In such an event, the current committers plan to continue working on the project on their own time, though the progress will likely be slower. We plan to mitigate this risk by encouraging and recruiting additional committers. Since replicated log is a very common technique useful for large number of applications, we believe many developers would like to join and contribute to the project.
>
> === Inexperience with Open Source ===
> The initial committers include veteran Apache Members (Committers, PMC Members and Apache Members) and other developers who have varying degrees of experience with open source projects. All have been involved with source code that has been released under an open source license, and also have experience developing code with an open source development process.
>
> === Homogenous Developers ===
> The initial list of committers include senior Apache developers from multiple different organizations including Hortonworks, OfferUp, Intel, Uber etc. We believe this project will benefit many different projects and therefore more and more diverse set of developers will join over time.
>
> === Reliance on Salaried Developers ===
> It is expected that Concur development will occur on both salaried time and on volunteer time, after hours. The majority of initial committers are paid by their employer to contribute to this project. However, they are all passionate about the project, and we are confident that the project will continue even if no salaried developers contribute to the project. We are committed to recruiting additional committers including non-salaried developers.
>
> === Relationships with Other Apache Products ===
> Most of the initial developers are active participants in Apache Hadoop community and we expect adoption of Concur by Hadoop Community. However, Concur is a generic RAFT library and we will strive to take it to more general adoption. This project depends on Apache Maven, Apache Commons {collections,configuration,io} and Apache Hadoop.
>
> == Initial Source ==
> https://github.com/hortonworks/concur
>
> == External Dependencies ==
> * Apache Maven
> * Apache Commons {collections,configuration,io}
> * Apache Hadoop
> * Netty (Apache 2.0)
> * Guava (Apache 2.0)
> * Protobuf (Google 2014)
> * gRPC (Google 2015)
> * slf4j (MIT License)
> * Mockito (MIT)
> * junit (EPL 1.0)
> * FindBugs (LGPL)
>
>
> == Required Resources ==
>
> === Mailing List ===
> * concur-private
> * concur-dev
> * concur-user
> * concur-issues
>
> === Git Repository ===
> Git is the preferred source control system: git://git.apache.org/concur
>
> === Issue Tracking ===
> JIRA Concur (Concur)
>
>
> == Sponsors ==
>
> === Champion ===
> Jitendra Pandey (jitendra)
>
> === Nominated Mentors ===
> * Chris Nauroth (cnauroth)
> * Devaraj Das (devaraj)
> * Jakob Homan (jghoman)
> * Uma Maheswara Rao G (umamahesh)
>
>
> === Sponsoring Entity ===
> Incubator PMC
>
>
>
> == Reference ==
> * [1] Diego Ongaro and John Ousterhout. 2014. In search of an understandable consensus algorithm. In Proceedings of the 2014 USENIX conference on USENIX Annual Technical Conference (USENIX ATC'14), Garth Gibson and Nickolai Zeldovich (Eds.). USENIX Association, Berkeley, CA, USA, 305-320.
> * [2] https://raft.github.io/
> * [3] Apache DistributedLog (incubating) http://distributedlog.incubator.apache.org/
>
>
>
>
>
>
>
>
--
Jean-Baptiste Onofr
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org