You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by nitin sharma <ku...@gmail.com> on 2015/08/27 20:28:43 UTC
Storm -- Standby Datacenter for Failover
Hi All,
I have been asked how can we setup a failover datacenter for Storm, so that
if by any chance (lets say due to power outages) the complete Active
datacenter becomes offline then Standby Datacenter can become alive and
process the real time feeds.
Does anyone have faced this kind of scenario? what are the best practice to
follow?
Regards,
Nitin Kumar Sharma.
Re: Storm -- Standby Datacenter for Failover
Posted by ri...@chartbeat.com.
Sounds like a good plan.
> On Aug 29, 2015, at 1:20 AM, Kishore Senji <ks...@gmail.com> wrote:
>
> http://zookeeper.apache.org/doc/trunk/zookeeperObservers.html
>
> We can have a DC of observers as stand by and then have a DC of Storm on stand by configured to the local DC.
>
> Once the primary DC fails then the Observers in another DC can be changed as Participants and the stand by Storm cluster can deploy the topology. This can be automated if appropriate triggers are available to detect a DC failure.
>> On Fri, Aug 28, 2015 at 11:56 AM Rick Mangi <ri...@chartbeat.com> wrote:
>> You can, but that could potentially cause a major bottleneck and you would need at least 6 nodes to handle 1/2 of them dying at once. With zookeeper, adding more nodes can cause slow performance. I’d love to hear someone tell me I’m wrong and how to better manage it :)
>>
>> I’ve actually been working on writing our storm (non-trident) offsets from kafka to a separate zookeeper cluster from the one kafka uses to take the load off of our primary zookeeper cluster.
>>
>>
>>
>>> On Aug 28, 2015, at 1:41 PM, Susheel Kumar Gadalay <sk...@gmail.com> wrote:
>>>
>>> Can't you have zookeeper spawning data centers if there is connectivity.
>>>
>>> Enlarge zookeeper to more than 3 nodes and extra nodes from other data
>>> centers and
>>> also minimum replication to be set.
>>>
>>>
>>>> On 8/28/15, Rick Mangi <ri...@chartbeat.com> wrote:
>>>> As long as you are replicating kafka and your offset checkpoints in the
>>>> secondary datacenter you should be able to do this. The offsets into kafka
>>>> from your spout tell you where you are in your processing (as long as your
>>>> topology is acking).
>>>>
>>>> That said, if those offsets are in zookeeper you will need to find a way to
>>>> replicate those to your failover datacenter.
>>>>
>>>>> On Aug 27, 2015, at 3:08 PM, nitin sharma <ku...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> hi Andrew,
>>>>>
>>>>> our setup includes Kafka --> Storm --> Cassandra ..
>>>>>
>>>>> we want to have a backup cluster for Storm, so that in case of any outage
>>>>> the backup server (topology) picks up from where Active server topology
>>>>> stopped and process the events and sends them to Cassandra.
>>>>>
>>>>>
>>>>> Regards,
>>>>> Nitin Kumar Sharma.
>>>>>
>>>>>
>>>>> On Thu, Aug 27, 2015 at 3:04 PM, Andrew Xor <andreas.grammenos@gmail.com
>>>>> <ma...@gmail.com>> wrote:
>>>>> I can try and give you some ideas, but start by first telling us how
>>>>> you've setup your infrastructure a bit so we can better help you...
>>>>>
>>>>> Kindly yours,
>>>>>
>>>>> Andrew Grammenos
>>>>>
>>>>> -- PGP PKey --
>>>>> <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>
>>>>> https://www.dropbox.com/s/yxvycjvlsc111bh/pgpsig.txt
>>>>> <https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt>
>>>>>
>>>>> On Thu, Aug 27, 2015 at 9:28 PM, nitin sharma <kumarsharma.nitin@gmail.com
>>>>> <ma...@gmail.com>> wrote:
>>>>> Hi All,
>>>>>
>>>>> I have been asked how can we setup a failover datacenter for Storm, so
>>>>> that if by any chance (lets say due to power outages) the complete Active
>>>>> datacenter becomes offline then Standby Datacenter can become alive and
>>>>> process the real time feeds.
>>>>>
>>>>> Does anyone have faced this kind of scenario? what are the best practice
>>>>> to follow?
>>>>>
>>>>> Regards,
>>>>> Nitin Kumar Sharma.
Re: Storm -- Standby Datacenter for Failover
Posted by Kishore Senji <ks...@gmail.com>.
http://zookeeper.apache.org/doc/trunk/zookeeperObservers.html
We can have a DC of observers as stand by and then have a DC of Storm on
stand by configured to the local DC.
Once the primary DC fails then the Observers in another DC can be changed
as Participants and the stand by Storm cluster can deploy the topology.
This can be automated if appropriate triggers are available to detect a DC
failure.
On Fri, Aug 28, 2015 at 11:56 AM Rick Mangi <ri...@chartbeat.com> wrote:
> You can, but that could potentially cause a major bottleneck and you would
> need at least 6 nodes to handle 1/2 of them dying at once. With zookeeper,
> adding more nodes can cause slow performance. I’d love to hear someone tell
> me I’m wrong and how to better manage it :)
>
> I’ve actually been working on writing our storm (non-trident) offsets from
> kafka to a separate zookeeper cluster from the one kafka uses to take the
> load off of our primary zookeeper cluster.
>
>
>
> On Aug 28, 2015, at 1:41 PM, Susheel Kumar Gadalay <sk...@gmail.com>
> wrote:
>
> Can't you have zookeeper spawning data centers if there is connectivity.
>
> Enlarge zookeeper to more than 3 nodes and extra nodes from other data
> centers and
> also minimum replication to be set.
>
>
> On 8/28/15, Rick Mangi <ri...@chartbeat.com> wrote:
>
> As long as you are replicating kafka and your offset checkpoints in the
> secondary datacenter you should be able to do this. The offsets into kafka
> from your spout tell you where you are in your processing (as long as your
> topology is acking).
>
> That said, if those offsets are in zookeeper you will need to find a way to
> replicate those to your failover datacenter.
>
> On Aug 27, 2015, at 3:08 PM, nitin sharma <ku...@gmail.com>
> wrote:
>
> hi Andrew,
>
> our setup includes Kafka --> Storm --> Cassandra ..
>
> we want to have a backup cluster for Storm, so that in case of any outage
> the backup server (topology) picks up from where Active server topology
> stopped and process the events and sends them to Cassandra.
>
>
> Regards,
> Nitin Kumar Sharma.
>
>
> On Thu, Aug 27, 2015 at 3:04 PM, Andrew Xor <andreas.grammenos@gmail.com
> <mailto:andreas.grammenos@gmail.com <an...@gmail.com>>> wrote:
> I can try and give you some ideas, but start by first telling us how
> you've setup your infrastructure a bit so we can better help you...
>
> Kindly yours,
>
> Andrew Grammenos
>
> -- PGP PKey --
> <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>
> https://www.dropbox.com/s/yxvycjvlsc111bh/pgpsig.txt
> <https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt>
>
> On Thu, Aug 27, 2015 at 9:28 PM, nitin sharma <kumarsharma.nitin@gmail.com
> <mailto:kumarsharma.nitin@gmail.com <ku...@gmail.com>>> wrote:
> Hi All,
>
> I have been asked how can we setup a failover datacenter for Storm, so
> that if by any chance (lets say due to power outages) the complete Active
> datacenter becomes offline then Standby Datacenter can become alive and
> process the real time feeds.
>
> Does anyone have faced this kind of scenario? what are the best practice
> to follow?
>
> Regards,
> Nitin Kumar Sharma.
>
>
>
Re: Storm -- Standby Datacenter for Failover
Posted by Rick Mangi <ri...@chartbeat.com>.
You can, but that could potentially cause a major bottleneck and you would need at least 6 nodes to handle 1/2 of them dying at once. With zookeeper, adding more nodes can cause slow performance. I’d love to hear someone tell me I’m wrong and how to better manage it :)
I’ve actually been working on writing our storm (non-trident) offsets from kafka to a separate zookeeper cluster from the one kafka uses to take the load off of our primary zookeeper cluster.
> On Aug 28, 2015, at 1:41 PM, Susheel Kumar Gadalay <sk...@gmail.com> wrote:
>
> Can't you have zookeeper spawning data centers if there is connectivity.
>
> Enlarge zookeeper to more than 3 nodes and extra nodes from other data
> centers and
> also minimum replication to be set.
>
>
> On 8/28/15, Rick Mangi <rick@chartbeat.com <ma...@chartbeat.com>> wrote:
>> As long as you are replicating kafka and your offset checkpoints in the
>> secondary datacenter you should be able to do this. The offsets into kafka
>> from your spout tell you where you are in your processing (as long as your
>> topology is acking).
>>
>> That said, if those offsets are in zookeeper you will need to find a way to
>> replicate those to your failover datacenter.
>>
>>> On Aug 27, 2015, at 3:08 PM, nitin sharma <ku...@gmail.com>
>>> wrote:
>>>
>>> hi Andrew,
>>>
>>> our setup includes Kafka --> Storm --> Cassandra ..
>>>
>>> we want to have a backup cluster for Storm, so that in case of any outage
>>> the backup server (topology) picks up from where Active server topology
>>> stopped and process the events and sends them to Cassandra.
>>>
>>>
>>> Regards,
>>> Nitin Kumar Sharma.
>>>
>>>
>>> On Thu, Aug 27, 2015 at 3:04 PM, Andrew Xor <andreas.grammenos@gmail.com
>>> <mailto:andreas.grammenos@gmail.com <ma...@gmail.com>>> wrote:
>>> I can try and give you some ideas, but start by first telling us how
>>> you've setup your infrastructure a bit so we can better help you...
>>>
>>> Kindly yours,
>>>
>>> Andrew Grammenos
>>>
>>> -- PGP PKey --
>>> <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>>
>>> https://www.dropbox.com/s/yxvycjvlsc111bh/pgpsig.txt <https://www.dropbox.com/s/yxvycjvlsc111bh/pgpsig.txt>
>>> <https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt <https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt>>
>>>
>>> On Thu, Aug 27, 2015 at 9:28 PM, nitin sharma <kumarsharma.nitin@gmail.com <ma...@gmail.com>
>>> <mailto:kumarsharma.nitin@gmail.com <ma...@gmail.com>>> wrote:
>>> Hi All,
>>>
>>> I have been asked how can we setup a failover datacenter for Storm, so
>>> that if by any chance (lets say due to power outages) the complete Active
>>> datacenter becomes offline then Standby Datacenter can become alive and
>>> process the real time feeds.
>>>
>>> Does anyone have faced this kind of scenario? what are the best practice
>>> to follow?
>>>
>>> Regards,
>>> Nitin Kumar Sharma.
Re: Storm -- Standby Datacenter for Failover
Posted by Susheel Kumar Gadalay <sk...@gmail.com>.
Can't you have zookeeper spawning data centers if there is connectivity.
Enlarge zookeeper to more than 3 nodes and extra nodes from other data
centers and
also minimum replication to be set.
On 8/28/15, Rick Mangi <ri...@chartbeat.com> wrote:
> As long as you are replicating kafka and your offset checkpoints in the
> secondary datacenter you should be able to do this. The offsets into kafka
> from your spout tell you where you are in your processing (as long as your
> topology is acking).
>
> That said, if those offsets are in zookeeper you will need to find a way to
> replicate those to your failover datacenter.
>
>> On Aug 27, 2015, at 3:08 PM, nitin sharma <ku...@gmail.com>
>> wrote:
>>
>> hi Andrew,
>>
>> our setup includes Kafka --> Storm --> Cassandra ..
>>
>> we want to have a backup cluster for Storm, so that in case of any outage
>> the backup server (topology) picks up from where Active server topology
>> stopped and process the events and sends them to Cassandra.
>>
>>
>> Regards,
>> Nitin Kumar Sharma.
>>
>>
>> On Thu, Aug 27, 2015 at 3:04 PM, Andrew Xor <andreas.grammenos@gmail.com
>> <ma...@gmail.com>> wrote:
>> I can try and give you some ideas, but start by first telling us how
>> you've setup your infrastructure a bit so we can better help you...
>>
>> Kindly yours,
>>
>> Andrew Grammenos
>>
>> -- PGP PKey --
>> <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>
>> https://www.dropbox.com/s/yxvycjvlsc111bh/pgpsig.txt
>> <https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt>
>>
>> On Thu, Aug 27, 2015 at 9:28 PM, nitin sharma <kumarsharma.nitin@gmail.com
>> <ma...@gmail.com>> wrote:
>> Hi All,
>>
>> I have been asked how can we setup a failover datacenter for Storm, so
>> that if by any chance (lets say due to power outages) the complete Active
>> datacenter becomes offline then Standby Datacenter can become alive and
>> process the real time feeds.
>>
>> Does anyone have faced this kind of scenario? what are the best practice
>> to follow?
>>
>> Regards,
>> Nitin Kumar Sharma.
>>
>>
>>
>
>
Re: Storm -- Standby Datacenter for Failover
Posted by Rick Mangi <ri...@chartbeat.com>.
As long as you are replicating kafka and your offset checkpoints in the secondary datacenter you should be able to do this. The offsets into kafka from your spout tell you where you are in your processing (as long as your topology is acking).
That said, if those offsets are in zookeeper you will need to find a way to replicate those to your failover datacenter.
> On Aug 27, 2015, at 3:08 PM, nitin sharma <ku...@gmail.com> wrote:
>
> hi Andrew,
>
> our setup includes Kafka --> Storm --> Cassandra ..
>
> we want to have a backup cluster for Storm, so that in case of any outage the backup server (topology) picks up from where Active server topology stopped and process the events and sends them to Cassandra.
>
>
> Regards,
> Nitin Kumar Sharma.
>
>
> On Thu, Aug 27, 2015 at 3:04 PM, Andrew Xor <andreas.grammenos@gmail.com <ma...@gmail.com>> wrote:
> I can try and give you some ideas, but start by first telling us how you've setup your infrastructure a bit so we can better help you...
>
> Kindly yours,
>
> Andrew Grammenos
>
> -- PGP PKey --
> <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>
> https://www.dropbox.com/s/yxvycjvlsc111bh/pgpsig.txt <https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt>
>
> On Thu, Aug 27, 2015 at 9:28 PM, nitin sharma <kumarsharma.nitin@gmail.com <ma...@gmail.com>> wrote:
> Hi All,
>
> I have been asked how can we setup a failover datacenter for Storm, so that if by any chance (lets say due to power outages) the complete Active datacenter becomes offline then Standby Datacenter can become alive and process the real time feeds.
>
> Does anyone have faced this kind of scenario? what are the best practice to follow?
>
> Regards,
> Nitin Kumar Sharma.
>
>
>
Re: Storm -- Standby Datacenter for Failover
Posted by nitin sharma <ku...@gmail.com>.
hi Andrew,
our setup includes Kafka --> Storm --> Cassandra ..
we want to have a backup cluster for Storm, so that in case of any outage
the backup server (topology) picks up from where Active server topology
stopped and process the events and sends them to Cassandra.
Regards,
Nitin Kumar Sharma.
On Thu, Aug 27, 2015 at 3:04 PM, Andrew Xor <an...@gmail.com>
wrote:
> I can try and give you some ideas, but start by first telling us how
> you've setup your infrastructure a bit so we can better help you...
>
> Kindly yours,
>
> Andrew Grammenos
>
> -- PGP PKey --
> <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>
> https://www.dropbox.com/s/yxvycjvlsc111bh/pgpsig.txt
> <https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt>
>
> On Thu, Aug 27, 2015 at 9:28 PM, nitin sharma <kumarsharma.nitin@gmail.com
> > wrote:
>
>> Hi All,
>>
>> I have been asked how can we setup a failover datacenter for Storm, so
>> that if by any chance (lets say due to power outages) the complete Active
>> datacenter becomes offline then Standby Datacenter can become alive and
>> process the real time feeds.
>>
>> Does anyone have faced this kind of scenario? what are the best practice
>> to follow?
>>
>> Regards,
>> Nitin Kumar Sharma.
>>
>>
>
Re: Storm -- Standby Datacenter for Failover
Posted by Andrew Xor <an...@gmail.com>.
I can try and give you some ideas, but start by first telling us how you've
setup your infrastructure a bit so we can better help you...
Kindly yours,
Andrew Grammenos
-- PGP PKey --
<https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>
https://www.dropbox.com/s/yxvycjvlsc111bh/pgpsig.txt
<https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt>
On Thu, Aug 27, 2015 at 9:28 PM, nitin sharma <ku...@gmail.com>
wrote:
> Hi All,
>
> I have been asked how can we setup a failover datacenter for Storm, so
> that if by any chance (lets say due to power outages) the complete Active
> datacenter becomes offline then Standby Datacenter can become alive and
> process the real time feeds.
>
> Does anyone have faced this kind of scenario? what are the best practice
> to follow?
>
> Regards,
> Nitin Kumar Sharma.
>
>