You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Darsh <da...@gmail.com> on 2016/04/28 19:39:36 UTC
Distribute load across multiple storm clusters.
Hi,
Is it possible to to distribute load across 2 clusters if I deploy same
topology to 2 storm clusters with same spout id?
We have 2 separate storm clusters with 0.10.0 version of storm running in 2
different availability zones. We are using storm-kafka spout (with simple
consumer) to process data from kafka and external common zookeeper to store
the kafka offsets.Kafka Topic has 32 partitions. 8 executors(
*parallelism_hint*) for spout in each cluster.
Thanks
Darsh
Re: Distribute load across multiple storm clusters.
Posted by Darsh <da...@gmail.com>.
Hi Patrick,
Sorry I think I confused you with my setup. I have separate nimbus for each
storm cluster.
Setup I used for testing,
3 node zk cluster with one node in 3 availability zones.
storm cluster 1=nimbus and 3 supervisor in availability zone 1
storm cluster 2=nimbus and 3 supervisor in availability zone 2
Darsh
On Thu, Apr 28, 2016 at 10:10 PM, Patrick.Brinton <
Patrick.Brinton@target.com> wrote:
> Darsh,
> I am in a bit of a crush for a deployment and perf testing but I will ask
> my experts to take a look tomorrow. I will set up a single nimbus and 3
> zookeepers in one data center, then I will distribute the supervisors and
> see what happens. I think we will always be limited in how we back up
> nimbus but I think we should be able to share processing for supervisors
> across data centers.
>
> Keep in touch and let me know if this is really what you are looking to
> test. I have a lot of toys at the moment and this seems like a worth while
> test for everyone.
>
> Patrick
>
> Patrick Brinton Sr. Engineer SWLM | ¤Target | 7000 Target Parkway North
> | Brooklyn Park, MN 55445 | 612.599.6523* (ph) *|
> Patrick.Brinton@target.com <C3...@target.com>
>
> From: Darsh <da...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Thursday, April 28, 2016 at 5:29 PM
> To: "user@storm.apache.org" <us...@storm.apache.org>
> Subject: Re: Distribute load across multiple storm clusters.
> Patrick,
>
> Thank you for replying. I did try but load isn't distributed. Both
> clusters are processing all the events individually on the topic.
>
>
> Darsh
>
> On Thu, Apr 28, 2016 at 2:56 PM, Patrick.Brinton <
> Patrick.Brinton@target.com> wrote:
>
>> Darsh,
>> I have never tried but I have a setup where I could try. As long as you
>> point to the same zooKeeper I think it would work. Give it a try and let
>> me know if you hit issues. If you do I will mimic your setup and we should
>> be able to figure it out.
>>
>> Patrick
>>
>> Patrick Brinton Sr. Engineer SWLM | ¤Target | 7000 Target Parkway North
>> | Brooklyn Park, MN 55445 | 612.599.6523* (ph) *|
>> Patrick.Brinton@target.com <C3...@target.com>
>>
>> From: Darsh <da...@gmail.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: Thursday, April 28, 2016 at 12:39 PM
>> To: "user@storm.apache.org" <us...@storm.apache.org>
>> Subject: Distribute load across multiple storm clusters.
>>
>> Hi,
>>
>>
>> Is it possible to to distribute load across 2 clusters if I deploy same
>> topology to 2 storm clusters with same spout id?
>>
>>
>> We have 2 separate storm clusters with 0.10.0 version of storm running in
>> 2 different availability zones. We are using storm-kafka spout (with simple
>> consumer) to process data from kafka and external common zookeeper to store
>> the kafka offsets.Kafka Topic has 32 partitions. 8 executors(
>> *parallelism_hint*) for spout in each cluster.
>>
>>
>>
>>
>>
>>
>> Thanks
>>
>> Darsh
>>
>
>
>
> --
> Thanks
>
> Darsh
>
--
Thanks
Darsh
Re: Distribute load across multiple storm clusters.
Posted by "Patrick.Brinton" <Pa...@target.com>.
Darsh,
I am in a bit of a crush for a deployment and perf testing but I will ask my experts to take a look tomorrow. I will set up a single nimbus and 3 zookeepers in one data center, then I will distribute the supervisors and see what happens. I think we will always be limited in how we back up nimbus but I think we should be able to share processing for supervisors across data centers.
Keep in touch and let me know if this is really what you are looking to test. I have a lot of toys at the moment and this seems like a worth while test for everyone.
Patrick
Patrick Brinton Sr. Engineer SWLM | ¤Target | 7000 Target Parkway North | Brooklyn Park, MN 55445 | 612.599.6523 (ph) | Patrick.Brinton@target.com<ma...@target.com>
From: Darsh <da...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Thursday, April 28, 2016 at 5:29 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Distribute load across multiple storm clusters.
Patrick,
Thank you for replying. I did try but load isn't distributed. Both clusters are processing all the events individually on the topic.
Darsh
On Thu, Apr 28, 2016 at 2:56 PM, Patrick.Brinton <Pa...@target.com>> wrote:
Darsh,
I have never tried but I have a setup where I could try. As long as you point to the same zooKeeper I think it would work. Give it a try and let me know if you hit issues. If you do I will mimic your setup and we should be able to figure it out.
Patrick
Patrick Brinton Sr. Engineer SWLM | ¤Target | 7000 Target Parkway North | Brooklyn Park, MN 55445 | 612.599.6523<tel:612.599.6523> (ph) | Patrick.Brinton@target.com<ma...@target.com>
From: Darsh <da...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Thursday, April 28, 2016 at 12:39 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Distribute load across multiple storm clusters.
Hi,
Is it possible to to distribute load across 2 clusters if I deploy same topology to 2 storm clusters with same spout id?
We have 2 separate storm clusters with 0.10.0 version of storm running in 2 different availability zones. We are using storm-kafka spout (with simple consumer) to process data from kafka and external common zookeeper to store the kafka offsets.Kafka Topic has 32 partitions. 8 executors(parallelism_hint) for spout in each cluster.
Thanks
Darsh
--
Thanks
Darsh
Re: Distribute load across multiple storm clusters.
Posted by Darsh <da...@gmail.com>.
Patrick,
Thank you for replying. I did try but load isn't distributed. Both clusters
are processing all the events individually on the topic.
Darsh
On Thu, Apr 28, 2016 at 2:56 PM, Patrick.Brinton <Patrick.Brinton@target.com
> wrote:
> Darsh,
> I have never tried but I have a setup where I could try. As long as you
> point to the same zooKeeper I think it would work. Give it a try and let
> me know if you hit issues. If you do I will mimic your setup and we should
> be able to figure it out.
>
> Patrick
>
> Patrick Brinton Sr. Engineer SWLM | ¤Target | 7000 Target Parkway North
> | Brooklyn Park, MN 55445 | 612.599.6523* (ph) *|
> Patrick.Brinton@target.com <C3...@target.com>
>
> From: Darsh <da...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Thursday, April 28, 2016 at 12:39 PM
> To: "user@storm.apache.org" <us...@storm.apache.org>
> Subject: Distribute load across multiple storm clusters.
>
> Hi,
>
>
> Is it possible to to distribute load across 2 clusters if I deploy same
> topology to 2 storm clusters with same spout id?
>
>
> We have 2 separate storm clusters with 0.10.0 version of storm running in
> 2 different availability zones. We are using storm-kafka spout (with simple
> consumer) to process data from kafka and external common zookeeper to store
> the kafka offsets.Kafka Topic has 32 partitions. 8 executors(
> *parallelism_hint*) for spout in each cluster.
>
>
>
>
>
>
> Thanks
>
> Darsh
>
--
Thanks
Darsh
Re: Distribute load across multiple storm clusters.
Posted by "Patrick.Brinton" <Pa...@target.com>.
Darsh,
I have never tried but I have a setup where I could try. As long as you point to the same zooKeeper I think it would work. Give it a try and let me know if you hit issues. If you do I will mimic your setup and we should be able to figure it out.
Patrick
Patrick Brinton Sr. Engineer SWLM | ¤Target | 7000 Target Parkway North | Brooklyn Park, MN 55445 | 612.599.6523 (ph) | Patrick.Brinton@target.com<ma...@target.com>
From: Darsh <da...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Thursday, April 28, 2016 at 12:39 PM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Distribute load across multiple storm clusters.
Hi,
Is it possible to to distribute load across 2 clusters if I deploy same topology to 2 storm clusters with same spout id?
We have 2 separate storm clusters with 0.10.0 version of storm running in 2 different availability zones. We are using storm-kafka spout (with simple consumer) to process data from kafka and external common zookeeper to store the kafka offsets.Kafka Topic has 32 partitions. 8 executors(parallelism_hint) for spout in each cluster.
Thanks
Darsh