You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sling.apache.org by "Stefan Egli (JIRA)" <ji...@apache.org> on 2013/07/01 15:49:20 UTC
[jira] [Comment Edited] (SLING-2939) 3rd-party based implementation
of discovery.api
[ https://issues.apache.org/jira/browse/SLING-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13696787#comment-13696787 ]
Stefan Egli edited comment on SLING-2939 at 7/1/13 1:48 PM:
------------------------------------------------------------
Comparison of candidates
Messaging
*-JMS / java message service, could be used to communicate state between instances and elect leader etc
+ well established API
+ HA JMS exists
- no ootb support for liveliness support
- no ootb support for leader election
=> not suited at all
Messaging
JGroups
http://www.jgroups.org/
JGroups supports the concept of membership with which you'd get leader election for free
+ ootb support for liveliness detection
+ ootb support for leader election
- No central services, embedded by default
- no ootb support for property storage/propagation
- LGPL (plans for Apache Licence on the way)
=> Usable but would require property propagation to be implemented/added
Messaging
other / *-messaging
=> There are of course many other messaging based solutions (eg [Apache Kafka|http://kafka.apache.org/]) that could be candidates for a 3rd party solution. But in the end the discovery.api is not about messaging and thus a messaging solution is not the ideal fit probably.
Distributed Caches/in-memory grids
Hazelcast Community Edition
http://www.hazelcast.com/products-community.jsp
+ Supports distributed implementations for various java.util.concurrent objects
+ ootb support for liveliness detection
+ ootb support for property storage/propagation
- No central services, embedded by default
- no ootb support for leader election. has to be coded based on locks
- 'community edition'
- embedded mode only
=>| IMHO geared towards UDP/Multicast, but TCP also possible. Large, cross data-center deployments of Hazelcast via TCP is IMHO similar to ZooKeeper-embedded and less ideal, although possible to deploy.
Distributed Caches/in-memory grids
other / *-memory grid
=> There are other memory grid-like solutions out there (eg [Terracotta|http://terracotta.org/], [Coherence|http://www.oracle.com/technetwork/middleware/coherence/overview/index.html]) that could be candidates. And the delineation between such solutions and distributed coordination based solution is less clear than vs messaging. What can be said though that in the end, any such solution's main strength is distributed memory - and that's not what discovery.api requires exactly.
Distributed Coordination
Apache ZooKeeper
http://zookeeper.apache.org/
ZooKeeper is a centralized yet distributed/highly available configuration service which supports various distributed coordination patterns through _recipes_
+ ootb support for liveliness detection
+ ootb support for property storage/propagation
+ supports both dedicated and embedded modes
- no ootb support for leader election. has to be coded based on ephemeral nodes
=> While [ZooKeeper] fulfills the requirements, it is known for being tricky to deal with (exception handling, session-reconnects, leader election needs to be coded according to provided recipes). Can be deployed embedded or dedicated.
Distributed Coordination
Apache Curator
http://curator.incubator.apache.org/
Built on top of ZooKeeper, Curator provides default implementation of the recipes documented in ZooKeeper
+ ootb support for liveliness detection
+ ootb support for property storage/propagation
+ ootb support for leader election
+ supports both dedicated and embedded modes
=> Matches all requirements. Can be deployed embedded or dedicated.
Cluster Management
Apache Helix
http://helix.incubator.apache.org/
Built on top of ZooKeeper, Helix is a cluster management framework for handling distributed resources
=> Compared to Apache Curator, Helix has a different focus (management of resources in a cluster) while the discovery.api does not require any of Helix' additional features. In this direct comparison Apache Curator is clearly the better fit.
* A note on embedding ZooKeeper: ZooKeeper and its father Google Chubby were designed to be a centralized, dedicated, clustered service providing configuration and coordination to a large set of nodes. Typical deployments of ZooKeeper are 3, 5 or 7 replicated instances. While you +can+ embed ZooKeeper in an application (possible thanks to ZOOKEEPER-107 (https://issues.apache.org/jira/browse/ZOOKEEPER-107) this mode is IMHO less suited for large, distributed setups like in CQ/Granite case. Each Granite instance would contain an embedded ZooKeeper and talk to a set of other Granite instances through the ZooKeeper protocols. Especially when deployed accross data centers, the embedded mode is not optimal.
* Another note on embedding ZooKeeper: Apparently (http://zookeeper-user.578899.n2.nabble.com/Embedding-ZK-in-another-application-td4951714.html) there are/were places where the embedded ZooKeeper does a System.exit. This would be entirely unacceptable and needs to be analyzed if this voids the zookeeper-embedding (ZOOKEEPER-575 tracks the System.exit issue: https://issues.apache.org/jira/browse/ZOOKEEPER-575). The former article suggests not to use embedded ZooKeeper btw - this one: https://support.lucidworks.com/entries/23343101-Should-you-run-embedded-zookeeper- as well.
was (Author: egli):
h3. Comparison of candidates
|| Category || Name || Short Description || Pros \\ || Cons \\ || Conclusion \\ ||
| Messaging | [\*-JMS |http://en.wikipedia.org/wiki/Java_Message_Service]\\ | java message service. \\
could be used to communicate state between instances and elect leader etc \\ | * well established API
* HA JMS exists | * no ootb support for liveliness detection
* no ootb support for leader election | Not suited at all \\ |
| | [JGroups|http://www.jgroups.org/]\\ | JGroups supports the concept of membership with which you'd get leader election for free \\ | * ootb support for liveliness detection
* ootb support for leader election
* No central services, embedded by default | * no ootb support for property storage/propagation | Usable but would require property propagation to be implemented/added \\ |
| | \*-messaging | | | | There are of course many other messaging based solutions (eg [Apache Kafka|http://kafka.apache.org/]) that could be candidates for a 3rd party solution. But in the end the discovery.api is not about messaging and thus a messaging solution is not the ideal fit probably. \\ |
| Distributed Caches/In-memory grids \\ | [Hazelcast Community Edition|http://www.hazelcast.com/products-community.jsp]\\ | Supports distributed implementations for various java.util.concurrent objects \\ | * ootb support for liveliness detection
* ootb support for property storage/propagation
* No central services, embedded by default | * no ootb support for leader election. has to be coded based on locks
* 'community edition'
* embedded mode only | IMHO geared towards UDP/Multicast, but TCP also possible. \\
Large, cross data-center deployments of Hazelcast via TCP is IMHO similar to [ZooKeeper]\-embedded and less ideal, although possible to deploy. \\ |
| | \*-memory grid \\ | | | | There are other memory grid-like solutions out there (eg [Terracotta|http://terracotta.org/], [Coherence|http://www.oracle.com/technetwork/middleware/coherence/overview/index.html]) that could be candidates. And the delineation between such solutions and distributed coordination based solution is less clear than vs messaging. What can be said though that in the end, any such solution's main strength is distributed memory - and that's not what discovery.api requires exactly. \\ |
| Distributed Coordination \\ | [Apache ZooKeeper|http://zookeeper.apache.org/]\\ | [ZooKeeper] is a centralized yet distributed/highly available configuration service which supports various distributed coordination patterns through _recipes_ \\ | * ootb support for liveliness detection
* ootb support for property storage/propagation
* supports both dedicated and embedded modes | * no ootb support for leader election. has to be coded based on ephemeral nodes | While [ZooKeeper] fulfills the requirements, it is known for being tricky to deal with (exception handling, session-reconnects, leader election needs to be coded according to provided recipes). \\
Can be deployed embedded or dedicated. \\ |
| | [Apache Curator|http://curator.incubator.apache.org/]\\ | Built on top of [ZooKeeper], Curator provides default implementation of the recipes documented in [ZooKeeper]. \\ | * ootb support for liveliness detection
* ootb support for property storage/propagation
* ootb support for leader election
* supports both dedicated and embedded modes | | Matches all requirements. \\
Can be deployed embedded or dedicated. \\ |
| Cluster Management \\ | [Apache Helix|http://helix.incubator.apache.org/]\\ | Built on top of [ZooKeeper], Helix is a cluster management framework for handling distributed resources \\ | * similar to Apache Curator | * similar to Apache Curator | Compared to Apache Curator, Helix has a different focus (management of resources in a cluster) while the discovery.api does not require any of Helix' additional features. In this direct comparison Apache Curator is clearly the better fit. \\ |
* A note on embedding [ZooKeeper]: [ZooKeeper] and its father Google Chubby were designed to be a centralized, dedicated, clustered service providing configuration and coordination to a large set of nodes. Typical deployments of [ZooKeeper] are 3, 5 or 7 replicated instances. While you +can+ embed [ZooKeeper] in an application (possible thanks to [ZOOKEEPER-107|https://issues.apache.org/jira/browse/ZOOKEEPER-107]) this mode is IMHO less suited for large, distributed setups like in CQ/Granite case. Each Granite instance would contain an embedded [ZooKeeper] and talk to a set of other Granite instances through the [ZooKeeper] protocols. Especially when deployed accross data centers, the embedded mode is not optimal.
* Another note on embedding [ZooKeeper]: [Apparently|http://zookeeper-user.578899.n2.nabble.com/Embedding-ZK-in-another-application-td4951714.html] there are places where the embedded [ZooKeeper] does a System.exit. This would be entirely unacceptable and needs to be analyzed if this voids the zookeeper-embedding ([ZOOKEEPER-575 tracks the System.exit issue|https://issues.apache.org/jira/browse/ZOOKEEPER-575]. The former article suggests not to use embedded [ZooKeeper] btw - [this one|https://support.lucidworks.com/entries/23343101-Should-you-run-embedded-zookeeper-] as well.
* [Slides about ZAB : ZooKeeper Atomic Broadcast|https://cwiki.apache.org/confluence/download/attachments/24193445/zookeeper-internals.pdf?version=1&modificationDate=1295034038000]
> 3rd-party based implementation of discovery.api
> -----------------------------------------------
>
> Key: SLING-2939
> URL: https://issues.apache.org/jira/browse/SLING-2939
> Project: Sling
> Issue Type: Task
> Components: Extensions
> Affects Versions: Discovery API 1.0.0
> Reporter: Stefan Egli
> Assignee: Stefan Egli
>
> The Sling Discovery API introduces the abstraction of a topology which contains (Sling) clusters and instances, supports liveliness-detection, leader-election within a cluster and property-propagation between the instances. As a default and reference implementation a resource-based, OOTB implementation was created (org.apache.sling.discovery.impl).
> Pros and cons of the discovery.impl
> Although the discovery.impl supports everything required in discovery.api, it has a few limitations. Here's a list of pros and cons:
> Pros
> No additional software required (leverages repository for intra-cluster communication/storage and HTTP-REST calls for cross-cluster communication)
> Very small footprint
> Perfectly suited for a single clusters, instance and for small, rather stable hub-based topologies
> Cons
> Config-/deployment-limitations (aka embedded-limitation): connections between clusters are peer-to-peer and explicit. To span a topology, a number of instances must (be made) know (to) each other, changes in the topology typically requires config adjustments to guarantee high availability of the discovery service
> Except if a natural "hub cluster" exists that can serve as connection point for all "satellite clusters"
> Other than that, it is less suited for large and/or dynamic topologies
> Change propagation (for topology parts reported via connectors) is non-atomic and slow, hop-by-hop based
> No guarantee on order of TopologyEvents sent in individual instances - ie different instances might see different orders of TopologyEvents (ie changes in the topology) but eventually the topology is guaranteed to be consistent
> Robustness of discovery.impl wrt storm situations depends on robustness of underlying cluster (not a real negative but discovery.impl might in theory unveil repository bugs which would otherwise not have been a problem)
> Rather new, little tested code which might have issues with edge cases wrt network problems
> although partitioning-support is not a requirement per se, similar edge-cases might exist wrt network-delays/timing/crashes
> Reusing a suitable 3rd party library
> To provide an additional option as implementation of the discovery.api one idea is to use a suitable 3rd party library.
> Requirements
> The following is a list of requirements a 3rd party library must support:
> liveliness detection: detect whether an instance is up and running
> stable leader election within a cluster: stable describes the fact that a leader will remain leader until it leaves/crashes and no new, joining instance shall take over while a leader exists
> stable instance ordering: the list of instances within a cluster is ordered and stable, new, joining instances are put at the end of the list
> property propagation: propagate the properties provided within one instance to everybody in the topology. there are no timing requirements bound to this but the intention of this is not to be used as messaging but to announce config parameters to the topology
> support large, dynamic clusters: configuration of the new discovery implementation should be easy and support frequent changes in the (large) topology
> no single point of failure: this is obvious, there should of course be no single point of failure in the setup
> embedded or dedicated: this might be a hot topic: embedding a library has the advantages of not having to install anything additional. a dedicated service on the other hand requires additional handling in deployment. embedding implies a peer-to-peer setup: nodes communicate peer-to-peer rather than via a centralized service. this IMHO is a negative for large topologies which would typically be cross data-centers. hence a dedicated service could be seen as an advantage in the end.
> due to need for cross data-center deployments, the transport protocol must be TCP (or HTTP for that matter)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira