Helix and Hbase

[kishore g <g....@gmail.com>]

Hello,

I have been wanting to write a proposal on using Apache Helix for cluster
management in HBase but wanted to hear some thoughts/feedback on whether
such an exercise would be useful.

I understand that Hbase has built its own cluster management solution but
integrating with Helix can provide additional benefits such as

   - Multiple replicas per region. Assigning roles to each replica such as
   primary/secondary.
   - Handle fail over - promote secondary to master.
   - Expansion- Redistributes the load when new nodes are added.

In terms of architecture, Helix fits well with HBase architecture. Helix
controller is similar to HBase Master. Similar to Hbase it uses Zookeeper
to store the cluster state. We have built in many optimizations to make the
fail over fast and reliable.

The core philosophy behind Helix as been to abstract out cluster management
from core functionality and treat cluster management as first class
citizen. This allows systems to benefit from features developed at Helix
such as flapping detection, non trivial fault detection, chaos monkey,
throttling movement of data etc.
Apache Helix is currently used to power the core back end infrastructure
(data store, search & pub/sub) components at LinkedIn and is in production
for more than a year managing more than 1k machines.

Appreciate feedback/thoughts on this topic.

thanks,
Kishore G


Additional info:
Helix - http://helix.incubator.apache.org
SOCC Paper <http://www.slideshare.net/KishoreGopalakrishna/helix-onecol>
Reading material: Systems that use Helix
Espresso <http://www.slideshare.net/amywtang/espresso-20952131>
Databus<https://915bbc94-a-62cb3a1a-s-sites.googlegroups.com/site/acm2012socc/s18-das.pdf>