How to manage a cluster with 10k nodes (or 100k partitions) using Helix?

[梁成辉 <ke...@163.com>]

 Dear all,

I'm working on a project which support up to 10 thousand physical nodes(or 100 thousand partitions). 
All of this nodes will became the Participant of Helix.
Can Helix manage such a big cluster?

How to use Helix to deal with it?


Best regards,

Ken

Re: How to manage a cluster with 10k nodes (or 100k partitions) using Helix?

[kishore g <g....@gmail.com>]

Hi Ken,

The maximum we have seen in few thousands of nodes in one cluster and
around 30-40k partitions. I don't think we have tried 10 thousand nodes in
production. Here are few things you might want to deal with the scale.


   1. Need 5 or 7 zookeepers, each participant of Helix creates one
   connection to Zookeeper. Assuming even distribution this means 2k
   connections per zookeeper server.
   2. For large number of partitions you can use the new compression
   support added in 0.6.5
   3. Configure the resource to use group message feature, this will allow
   Helix to automatically group multiple state transitions messages into 1

There are more optimizations possible but that depends on the exact use
case.

thanks,
Kishore G


On Mon, Apr 13, 2015 at 7:06 AM, 梁成辉 <ke...@163.com> wrote:

>  Dear all,
>
>
> I'm working on a project which support up to 10 thousand physical nodes(or 100 thousand partitions).
> All of this nodes will became the Participant of Helix.
> Can Helix manage such a big cluster?
>
> How to use Helix to deal with it?
>
>
> Best regards,
>
> Ken
>
>
>
>