Sling integration with Etcd

[Timothée Maret <ti...@gmail.com>]

Hi,

I found some time to look closer to the Sling Discovery implementation on
AWS S3 proposal in [0].
Doing so, I stumbled upon Etcd [1] based on Raft [2] which seems to covers
exactly what I wanted to do.

Indeed, Etcd provides both persistent and temporal keys a well as
primitives to perform leader election.
Therefor, the Sling discovery API could be implemented on top of Etcd with
the notable addition of

* multi-level leader election (e.g. at deploymen level, region level, or
any arbitrary segmentation).
* persistent properties

Etcd would require to be deployed on dedicated HW (not embedded) but offers
HTTP API for Sling instances to communicate with it.

I have added this option in SLING-2939.

Has someone already implemented it ? If not, I think it would make much
more sense to implement this integration rather than the idea in [0] and I
would give it a go.

Regards,

Timothee


[0]
http://apache-sling.73963.n3.nabble.com/Sling-Discovery-implementation-on-AWS-S3-td4033160.html
[1] https://github.com/coreos/etcd
[2] https://raftconsensus.github.io

Re: Sling integration with Etcd

[Timothée Maret <ti...@gmail.com>]

Hi all,

Thanks for your input.

Yes, operations is something to consider, that's something we discuss
internally.
A small Go process also has its appeal when compared to a JVM deployment.
I'll give Etcd a go as it seems nobody has done it already.

Regards,

Timothee

2015-02-04 15:56 GMT+00:00 Bertrand Delacretaz <bd...@apache.org>:

> On Wed, Feb 4, 2015 at 4:45 PM, Bertrand Delacretaz
> <bd...@apache.org> wrote:
> > ...I haven't integrated etcd with the Sling discovery but I have used it
> > for my https://github.com/bdelacretaz/docker-sling-cluster prototype
> > and found it quite nice....
>
> Note also that Zookeeper running in a JVM is an advantage over etcd
> being AFAIK a Linux thing - "operations familiarity" might be a
> criteria to consider.
>
> -Bertrand
>

Re: Sling integration with Etcd

[Bertrand Delacretaz <bd...@apache.org>]

On Wed, Feb 4, 2015 at 4:45 PM, Bertrand Delacretaz
<bd...@apache.org> wrote:
> ...I haven't integrated etcd with the Sling discovery but I have used it
> for my https://github.com/bdelacretaz/docker-sling-cluster prototype
> and found it quite nice....

Note also that Zookeeper running in a JVM is an advantage over etcd
being AFAIK a Linux thing - "operations familiarity" might be a
criteria to consider.

-Bertrand

Re: Sling integration with Etcd

[Bruce Edge <br...@nextissuemedia.com>]

I made an Ubuntu package out of etcd for an unrelated project.
If anyone wants a copy let me know.

-Bruce

From: Bertrand Delacretaz <bd...@apache.org>
Reply-To: dev <de...@sling.apache.org>
Date: Wednesday, February 4, 2015 at 7:45 AM
To: dev <de...@sling.apache.org>
Subject: Re: Sling integration with Etcd


>Hi,
>
>On Wed, Feb 4, 2015 at 2:40 PM, Timothée Maret <ti...@gmail.com>
>wrote:
>>...Has someone already implemented it ?...
>
>I haven't integrated etcd with the Sling discovery but I have used it
>for my https://github.com/bdelacretaz/docker-sling-cluster prototype
>and found it quite nice.
>
>-Bertrand
>
>
>

Re: Sling integration with Etcd

[Bertrand Delacretaz <bd...@apache.org>]

Hi,

On Wed, Feb 4, 2015 at 2:40 PM, Timothée Maret <ti...@gmail.com> wrote:
> ...Has someone already implemented it ?...

I haven't integrated etcd with the Sling discovery but I have used it
for my https://github.com/bdelacretaz/docker-sling-cluster prototype
and found it quite nice.

-Bertrand

Re: Sling integration with Etcd

[Stefan Egli <st...@apache.org>]

Hi Timothée,

As discussed yesterday privately I absolutely support the idea of having
an additional, 3rd-party based implementation of discovery.api. Whether it
be etcd or zookeeper (or even something else) is up to feature
comparison/discussion/etc.. One thing I thought was mentioned is that "we
use ZooKeeper elsewhere already" and there might thus be synergies. But I
haven't done a direct comparison of etcd vs zookeeper so far, so don't
know if there is a clear winner anyway..

One thing to remember when implementing: one feature of discovery.api is
that it groups instances hooked to the same repository into the same
ClusterView and provides it with a stable (cluster)Id. In discovery.impl
that is achieved by chatting and voting via said repository. When using an
external mechanism, the fact that an instance belongs to a certain cluster
needs special treatment as well in one way or another: either by still
storing the id in the repository for example or via some configuration
mechanism maybe. (Just something to keep in mind..)

Cheers,
Stefan

On 2/4/15 2:40 PM, "Timothée Maret" <ti...@gmail.com> wrote:

>Hi,
>
>I found some time to look closer to the Sling Discovery implementation on
>AWS S3 proposal in [0].
>Doing so, I stumbled upon Etcd [1] based on Raft [2] which seems to covers
>exactly what I wanted to do.
>
>Indeed, Etcd provides both persistent and temporal keys a well as
>primitives to perform leader election.
>Therefor, the Sling discovery API could be implemented on top of Etcd with
>the notable addition of
>
>* multi-level leader election (e.g. at deploymen level, region level, or
>any arbitrary segmentation).
>* persistent properties
>
>Etcd would require to be deployed on dedicated HW (not embedded) but
>offers
>HTTP API for Sling instances to communicate with it.
>
>I have added this option in SLING-2939.
>
>Has someone already implemented it ? If not, I think it would make much
>more sense to implement this integration rather than the idea in [0] and I
>would give it a go.
>
>Regards,
>
>Timothee
>
>
>[0]
>http://apache-sling.73963.n3.nabble.com/Sling-Discovery-implementation-on-
>AWS-S3-td4033160.html
>[1] https://github.com/coreos/etcd
>[2] https://raftconsensus.github.io