You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@geode.apache.org by aashish choudhary <aa...@gmail.com> on 2019/04/10 18:01:17 UTC
Geode switch over to different data center
Hi,
We have a scenario where in we need to switch over to a different data
center automatically when any failure occurs in the existing cluster.
Any recommendations?
With best regards,
Ashish
Re: Geode switch over to different data center
Posted by aashish choudhary <aa...@gmail.com>.
Thanks, I will check it out.
With best regards,
Ashish
On Thu, Apr 16, 2020, 5:13 PM Gregory Green <gg...@pivotal.io> wrote:
> Hello Ashish,
>
> Hope all is well.
>
> FYI - here is a spring based example;
>
> https://github.com/vmwarepivotallabs/dataTx-geode-circuitbreaker-demo
>
> On Apr 16, 2020, at 6:48 AM, aashish choudhary <
> aashish.choudhary1@gmail.com> wrote:
>
>
> I think i don't need to create ClientCache again in the catch block as i
> am using separate pools for 2 clusters. Updated the code.
>
> https://github.com/userac/GeodeExample/blob/master/src/main/java/Example.java
> With Best Regards,
> Ashish
>
>
> On Thu, Apr 16, 2020 at 3:01 PM aashish choudhary <
> aashish.choudhary1@gmail.com> wrote:
>
>> Here is the simple code example that i tried for client side failover.
>> For my use case i will always try to connect to primary cluster first and
>> if it fails it should failover to secondary. I understand it may slow
>> things as i will be creating a connection on each request but there will be
>> no issue in getting response. Not sure if there is any better way to do
>> this. I should probably add something to stop bouncing back and forth if
>> both clusters are down.
>>
>>
>> https://github.com/userac/GeodeExample/blob/master/src/main/java/Example.java
>>
>>
>> Let me know what you think.
>>
>> With Best Regards,
>> Ashish
>>
>>
>> On Wed, Apr 15, 2020 at 4:32 PM aashish choudhary <
>> aashish.choudhary1@gmail.com> wrote:
>>
>>> Replying to old thread.
>>> Currently I am trying test client side failover programtically. Are
>>> there any working examples /best practices for this?
>>>
>>> Idea is to failover to different secondary cluster if the primary fails.
>>> I will try to share what i am able to do so far.
>>>
>>> But if anyone has a working example for this then it would be great.
>>>
>>> With best regards,
>>> Ashish
>>>
>>> On Tue, Apr 16, 2019, 9:37 PM Anthony Baker <ab...@pivotal.io> wrote:
>>>
>>>> The pattern I’ve seen used looks like this:
>>>>
>>>> User application (e.g. browser) >> Global load balancer >> Service
>>>> instances (e.g. tomcat) >> Geode cluster
>>>>
>>>> If you have the Geode clusters connected via WAN, you can redirect
>>>> traffic to different data centers by tweaking the LB config.
>>>>
>>>>
>>>> Anthony
>>>>
>>>>
>>>> On Apr 16, 2019, at 2:58 AM, aashish choudhary <
>>>> aashish.choudhary1@gmail.com> wrote:
>>>>
>>>> So with WAN we will be active/active all time. I agree hardest is to
>>>> figure out when data centers are actually down. We are evaluating multiple
>>>> approaches as of now.
>>>>
>>>> On that note would you recommend(possibility any since connection is
>>>> mostly tcp/ip) using some load balancer NGINX or something to handle data
>>>> center failure.
>>>>
>>>> With best regards,
>>>> Ashish
>>>>
>>>> On Tue, Apr 16, 2019, 12:54 AM Michael Stolz <ms...@pivotal.io> wrote:
>>>>
>>>>> We have come across these kinds of use-cases.
>>>>> The hardest part is figuring out that one of the data centers is
>>>>> ACTUALLY down.
>>>>>
>>>>> If you can work out a way to be active/active at all times and guard
>>>>> against update collisions by using data structures that protect themselves
>>>>> (e.g. CRDTs) that would make the whole thing a lot easier.
>>>>> --
>>>>> Mike Stolz
>>>>> Principal Engineer, GemFire Product Lead
>>>>> Mobile: +1-631-835-4771
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Apr 15, 2019 at 1:19 PM aashish choudhary <
>>>>> aashish.choudhary1@gmail.com> wrote:
>>>>>
>>>>>> Thanks Mike. Our use case is heavily reliant on Geode(no fallback to
>>>>>> database) and business expectation is that there will be no downtime to
>>>>>> consumer application because of complete failure on one data center. Which
>>>>>>
>>>>>> Have you came across such cases with Geode/Gemfire?
>>>>>>
>>>>>> Regarding catching those exceptions and making a switch I agree with
>>>>>> you that it would be tricky to make switch as you explained.
>>>>>>
>>>>>> Even with rolling restart there will be a downtime and some manual
>>>>>> steps will be required to accomplish that.
>>>>>>
>>>>>> With best regards,
>>>>>> Ashish
>>>>>>
>>>>>> On Thu, Apr 11, 2019, 10:18 PM Michael Stolz <ms...@pivotal.io>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes you can catch the exceptions for no locators available and no
>>>>>>> servers available.
>>>>>>> You will probably want to wait for a period of time after first
>>>>>>> seeing this, because the cluster might be restarting and will be back in
>>>>>>> just a minute or so.
>>>>>>>
>>>>>>> The switch-over can be tricky without just restarting your client.
>>>>>>>
>>>>>>> All saved references to everything having to do with ClientCache,
>>>>>>> Cache, Region, or anything else that communicates need to be forgotten and
>>>>>>> re-established.
>>>>>>> This can be particularly challenging if you are using a framework
>>>>>>> that might remember some of this stuff on your behalf.
>>>>>>> I have usually recommended rolling restart of the clients with the
>>>>>>> new locator addresses because it is sure to work and not have any
>>>>>>> hidden issues with calls in progress or subscriptions or anything like that.
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Mike Stolz
>>>>>>> Principal Engineer, GemFire Product Lead
>>>>>>> Mobile: +1-631-835-4771
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Apr 11, 2019 at 9:31 AM aashish choudhary <
>>>>>>> aashish.choudhary1@gmail.com> wrote:
>>>>>>>
>>>>>>>> Thanks Mike.
>>>>>>>>
>>>>>>>> Yes we are using wan replication. We want the switch to be an
>>>>>>>> automatic step. As soon as prod cluster fails we need to switch to cob
>>>>>>>> without any restart of the client application.
>>>>>>>>
>>>>>>>> One way we are thinking of is probably catching those locator not
>>>>>>>> available sort of exception and then make a switch. Any thoughts?
>>>>>>>>
>>>>>>>>
>>>>>>>> With best regards,
>>>>>>>> Ashish
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Apr 11, 2019, 1:42 AM Michael Stolz <ms...@pivotal.io>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> If the data centers are far apart you will want to use the
>>>>>>>>> bi-directional GemFire WAN Gateway to replicate between clusters.
>>>>>>>>>
>>>>>>>>> The trickiest part is figuring out when to switch. If you already
>>>>>>>>> have a mechanism for that then that's great.
>>>>>>>>>
>>>>>>>>> Once you know for sure you want to switch, the easiest way is to
>>>>>>>>> install a gemfire.properties file on the client machines that points to the
>>>>>>>>> locators in the other data center and restart the clients.
>>>>>>>>>
>>>>>>>>> There is a programmatic way to do it but is a lot more code and
>>>>>>>>> work than this way.
>>>>>>>>>
>>>>>>>>> Feel free to ask any additional questions here.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Mike Stolz
>>>>>>>>> Principal Engineer, GemFire Product Lead
>>>>>>>>> Mobile: +1-631-835-4771
>>>>>>>>>
>>>>>>>>> On Wed, Apr 10, 2019, 2:01 PM aashish choudhary <
>>>>>>>>> aashish.choudhary1@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> We have a scenario where in we need to switch over to a different
>>>>>>>>>> data center automatically when any failure occurs in the existing cluster.
>>>>>>>>>>
>>>>>>>>>> Any recommendations?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> With best regards,
>>>>>>>>>> Ashish
>>>>>>>>>>
>>>>>>>>>
>>>>
Re: Geode switch over to different data center
Posted by Gregory Green <gg...@pivotal.io>.
Hello Ashish,
Hope all is well.
FYI - here is a spring based example;
https://github.com/vmwarepivotallabs/dataTx-geode-circuitbreaker-demo
> On Apr 16, 2020, at 6:48 AM, aashish choudhary <aa...@gmail.com> wrote:
>
>
> I think i don't need to create ClientCache again in the catch block as i am using separate pools for 2 clusters. Updated the code.
> https://github.com/userac/GeodeExample/blob/master/src/main/java/Example.java
> With Best Regards,
> Ashish
>
>
>> On Thu, Apr 16, 2020 at 3:01 PM aashish choudhary <aa...@gmail.com> wrote:
>> Here is the simple code example that i tried for client side failover. For my use case i will always try to connect to primary cluster first and if it fails it should failover to secondary. I understand it may slow things as i will be creating a connection on each request but there will be no issue in getting response. Not sure if there is any better way to do this. I should probably add something to stop bouncing back and forth if both clusters are down.
>>
>> https://github.com/userac/GeodeExample/blob/master/src/main/java/Example.java
>>
>> Let me know what you think.
>>
>> With Best Regards,
>> Ashish
>>
>>
>>> On Wed, Apr 15, 2020 at 4:32 PM aashish choudhary <aa...@gmail.com> wrote:
>>> Replying to old thread.
>>> Currently I am trying test client side failover programtically. Are there any working examples /best practices for this?
>>>
>>> Idea is to failover to different secondary cluster if the primary fails. I will try to share what i am able to do so far.
>>>
>>> But if anyone has a working example for this then it would be great.
>>>
>>> With best regards,
>>> Ashish
>>>
>>>> On Tue, Apr 16, 2019, 9:37 PM Anthony Baker <ab...@pivotal.io> wrote:
>>>> The pattern I’ve seen used looks like this:
>>>>
>>>> User application (e.g. browser) >> Global load balancer >> Service instances (e.g. tomcat) >> Geode cluster
>>>>
>>>> If you have the Geode clusters connected via WAN, you can redirect traffic to different data centers by tweaking the LB config.
>>>>
>>>>
>>>> Anthony
>>>>
>>>>
>>>>> On Apr 16, 2019, at 2:58 AM, aashish choudhary <aa...@gmail.com> wrote:
>>>>>
>>>>> So with WAN we will be active/active all time. I agree hardest is to figure out when data centers are actually down. We are evaluating multiple approaches as of now.
>>>>>
>>>>> On that note would you recommend(possibility any since connection is mostly tcp/ip) using some load balancer NGINX or something to handle data center failure.
>>>>>
>>>>> With best regards,
>>>>> Ashish
>>>>>
>>>>>> On Tue, Apr 16, 2019, 12:54 AM Michael Stolz <ms...@pivotal.io> wrote:
>>>>>> We have come across these kinds of use-cases.
>>>>>> The hardest part is figuring out that one of the data centers is ACTUALLY down.
>>>>>>
>>>>>> If you can work out a way to be active/active at all times and guard against update collisions by using data structures that protect themselves (e.g. CRDTs) that would make the whole thing a lot easier.
>>>>>> --
>>>>>> Mike Stolz
>>>>>> Principal Engineer, GemFire Product Lead
>>>>>> Mobile: +1-631-835-4771
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Mon, Apr 15, 2019 at 1:19 PM aashish choudhary <aa...@gmail.com> wrote:
>>>>>>> Thanks Mike. Our use case is heavily reliant on Geode(no fallback to database) and business expectation is that there will be no downtime to consumer application because of complete failure on one data center. Which
>>>>>>>
>>>>>>> Have you came across such cases with Geode/Gemfire?
>>>>>>>
>>>>>>> Regarding catching those exceptions and making a switch I agree with you that it would be tricky to make switch as you explained.
>>>>>>>
>>>>>>> Even with rolling restart there will be a downtime and some manual steps will be required to accomplish that.
>>>>>>>
>>>>>>> With best regards,
>>>>>>> Ashish
>>>>>>>
>>>>>>>> On Thu, Apr 11, 2019, 10:18 PM Michael Stolz <ms...@pivotal.io> wrote:
>>>>>>>> Yes you can catch the exceptions for no locators available and no servers available.
>>>>>>>> You will probably want to wait for a period of time after first seeing this, because the cluster might be restarting and will be back in just a minute or so.
>>>>>>>>
>>>>>>>> The switch-over can be tricky without just restarting your client.
>>>>>>>>
>>>>>>>> All saved references to everything having to do with ClientCache, Cache, Region, or anything else that communicates need to be forgotten and re-established.
>>>>>>>> This can be particularly challenging if you are using a framework that might remember some of this stuff on your behalf.
>>>>>>>> I have usually recommended rolling restart of the clients with the new locator addresses because it is sure to work and not have any hidden issues with calls in progress or subscriptions or anything like that.
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Mike Stolz
>>>>>>>> Principal Engineer, GemFire Product Lead
>>>>>>>> Mobile: +1-631-835-4771
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Thu, Apr 11, 2019 at 9:31 AM aashish choudhary <aa...@gmail.com> wrote:
>>>>>>>>> Thanks Mike.
>>>>>>>>>
>>>>>>>>> Yes we are using wan replication. We want the switch to be an automatic step. As soon as prod cluster fails we need to switch to cob without any restart of the client application.
>>>>>>>>>
>>>>>>>>> One way we are thinking of is probably catching those locator not available sort of exception and then make a switch. Any thoughts?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> With best regards,
>>>>>>>>> Ashish
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Thu, Apr 11, 2019, 1:42 AM Michael Stolz <ms...@pivotal.io> wrote:
>>>>>>>>>> If the data centers are far apart you will want to use the bi-directional GemFire WAN Gateway to replicate between clusters.
>>>>>>>>>>
>>>>>>>>>> The trickiest part is figuring out when to switch. If you already have a mechanism for that then that's great.
>>>>>>>>>>
>>>>>>>>>> Once you know for sure you want to switch, the easiest way is to install a gemfire.properties file on the client machines that points to the locators in the other data center and restart the clients.
>>>>>>>>>>
>>>>>>>>>> There is a programmatic way to do it but is a lot more code and work than this way.
>>>>>>>>>>
>>>>>>>>>> Feel free to ask any additional questions here.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Mike Stolz
>>>>>>>>>> Principal Engineer, GemFire Product Lead
>>>>>>>>>> Mobile: +1-631-835-4771
>>>>>>>>>>
>>>>>>>>>>> On Wed, Apr 10, 2019, 2:01 PM aashish choudhary <aa...@gmail.com> wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> We have a scenario where in we need to switch over to a different data center automatically when any failure occurs in the existing cluster.
>>>>>>>>>>>
>>>>>>>>>>> Any recommendations?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> With best regards,
>>>>>>>>>>> Ashish
>>>>
Re: Geode switch over to different data center
Posted by aashish choudhary <aa...@gmail.com>.
I think i don't need to create ClientCache again in the catch block as i am
using separate pools for 2 clusters. Updated the code.
https://github.com/userac/GeodeExample/blob/master/src/main/java/Example.java
With Best Regards,
Ashish
On Thu, Apr 16, 2020 at 3:01 PM aashish choudhary <
aashish.choudhary1@gmail.com> wrote:
> Here is the simple code example that i tried for client side failover. For
> my use case i will always try to connect to primary cluster first and if it
> fails it should failover to secondary. I understand it may slow things as i
> will be creating a connection on each request but there will be no issue in
> getting response. Not sure if there is any better way to do this. I should
> probably add something to stop bouncing back and forth if both clusters are
> down.
>
>
> https://github.com/userac/GeodeExample/blob/master/src/main/java/Example.java
>
>
> Let me know what you think.
>
> With Best Regards,
> Ashish
>
>
> On Wed, Apr 15, 2020 at 4:32 PM aashish choudhary <
> aashish.choudhary1@gmail.com> wrote:
>
>> Replying to old thread.
>> Currently I am trying test client side failover programtically. Are there
>> any working examples /best practices for this?
>>
>> Idea is to failover to different secondary cluster if the primary fails.
>> I will try to share what i am able to do so far.
>>
>> But if anyone has a working example for this then it would be great.
>>
>> With best regards,
>> Ashish
>>
>> On Tue, Apr 16, 2019, 9:37 PM Anthony Baker <ab...@pivotal.io> wrote:
>>
>>> The pattern I’ve seen used looks like this:
>>>
>>> User application (e.g. browser) >> Global load balancer >> Service
>>> instances (e.g. tomcat) >> Geode cluster
>>>
>>> If you have the Geode clusters connected via WAN, you can redirect
>>> traffic to different data centers by tweaking the LB config.
>>>
>>>
>>> Anthony
>>>
>>>
>>> On Apr 16, 2019, at 2:58 AM, aashish choudhary <
>>> aashish.choudhary1@gmail.com> wrote:
>>>
>>> So with WAN we will be active/active all time. I agree hardest is to
>>> figure out when data centers are actually down. We are evaluating multiple
>>> approaches as of now.
>>>
>>> On that note would you recommend(possibility any since connection is
>>> mostly tcp/ip) using some load balancer NGINX or something to handle data
>>> center failure.
>>>
>>> With best regards,
>>> Ashish
>>>
>>> On Tue, Apr 16, 2019, 12:54 AM Michael Stolz <ms...@pivotal.io> wrote:
>>>
>>>> We have come across these kinds of use-cases.
>>>> The hardest part is figuring out that one of the data centers is
>>>> ACTUALLY down.
>>>>
>>>> If you can work out a way to be active/active at all times and guard
>>>> against update collisions by using data structures that protect themselves
>>>> (e.g. CRDTs) that would make the whole thing a lot easier.
>>>> --
>>>> Mike Stolz
>>>> Principal Engineer, GemFire Product Lead
>>>> Mobile: +1-631-835-4771
>>>>
>>>>
>>>>
>>>> On Mon, Apr 15, 2019 at 1:19 PM aashish choudhary <
>>>> aashish.choudhary1@gmail.com> wrote:
>>>>
>>>>> Thanks Mike. Our use case is heavily reliant on Geode(no fallback to
>>>>> database) and business expectation is that there will be no downtime to
>>>>> consumer application because of complete failure on one data center. Which
>>>>>
>>>>> Have you came across such cases with Geode/Gemfire?
>>>>>
>>>>> Regarding catching those exceptions and making a switch I agree with
>>>>> you that it would be tricky to make switch as you explained.
>>>>>
>>>>> Even with rolling restart there will be a downtime and some manual
>>>>> steps will be required to accomplish that.
>>>>>
>>>>> With best regards,
>>>>> Ashish
>>>>>
>>>>> On Thu, Apr 11, 2019, 10:18 PM Michael Stolz <ms...@pivotal.io>
>>>>> wrote:
>>>>>
>>>>>> Yes you can catch the exceptions for no locators available and no
>>>>>> servers available.
>>>>>> You will probably want to wait for a period of time after first
>>>>>> seeing this, because the cluster might be restarting and will be back in
>>>>>> just a minute or so.
>>>>>>
>>>>>> The switch-over can be tricky without just restarting your client.
>>>>>>
>>>>>> All saved references to everything having to do with ClientCache,
>>>>>> Cache, Region, or anything else that communicates need to be forgotten and
>>>>>> re-established.
>>>>>> This can be particularly challenging if you are using a framework
>>>>>> that might remember some of this stuff on your behalf.
>>>>>> I have usually recommended rolling restart of the clients with the
>>>>>> new locator addresses because it is sure to work and not have any
>>>>>> hidden issues with calls in progress or subscriptions or anything like that.
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Mike Stolz
>>>>>> Principal Engineer, GemFire Product Lead
>>>>>> Mobile: +1-631-835-4771
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Apr 11, 2019 at 9:31 AM aashish choudhary <
>>>>>> aashish.choudhary1@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks Mike.
>>>>>>>
>>>>>>> Yes we are using wan replication. We want the switch to be an
>>>>>>> automatic step. As soon as prod cluster fails we need to switch to cob
>>>>>>> without any restart of the client application.
>>>>>>>
>>>>>>> One way we are thinking of is probably catching those locator not
>>>>>>> available sort of exception and then make a switch. Any thoughts?
>>>>>>>
>>>>>>>
>>>>>>> With best regards,
>>>>>>> Ashish
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Apr 11, 2019, 1:42 AM Michael Stolz <ms...@pivotal.io>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> If the data centers are far apart you will want to use the
>>>>>>>> bi-directional GemFire WAN Gateway to replicate between clusters.
>>>>>>>>
>>>>>>>> The trickiest part is figuring out when to switch. If you already
>>>>>>>> have a mechanism for that then that's great.
>>>>>>>>
>>>>>>>> Once you know for sure you want to switch, the easiest way is to
>>>>>>>> install a gemfire.properties file on the client machines that points to the
>>>>>>>> locators in the other data center and restart the clients.
>>>>>>>>
>>>>>>>> There is a programmatic way to do it but is a lot more code and
>>>>>>>> work than this way.
>>>>>>>>
>>>>>>>> Feel free to ask any additional questions here.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Mike Stolz
>>>>>>>> Principal Engineer, GemFire Product Lead
>>>>>>>> Mobile: +1-631-835-4771
>>>>>>>>
>>>>>>>> On Wed, Apr 10, 2019, 2:01 PM aashish choudhary <
>>>>>>>> aashish.choudhary1@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> We have a scenario where in we need to switch over to a different
>>>>>>>>> data center automatically when any failure occurs in the existing cluster.
>>>>>>>>>
>>>>>>>>> Any recommendations?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> With best regards,
>>>>>>>>> Ashish
>>>>>>>>>
>>>>>>>>
>>>
Re: Geode switch over to different data center
Posted by aashish choudhary <aa...@gmail.com>.
Here is the simple code example that i tried for client side failover. For
my use case i will always try to connect to primary cluster first and if it
fails it should failover to secondary. I understand it may slow things as i
will be creating a connection on each request but there will be no issue in
getting response. Not sure if there is any better way to do this. I should
probably add something to stop bouncing back and forth if both clusters are
down.
https://github.com/userac/GeodeExample/blob/master/src/main/java/Example.java
Let me know what you think.
With Best Regards,
Ashish
On Wed, Apr 15, 2020 at 4:32 PM aashish choudhary <
aashish.choudhary1@gmail.com> wrote:
> Replying to old thread.
> Currently I am trying test client side failover programtically. Are there
> any working examples /best practices for this?
>
> Idea is to failover to different secondary cluster if the primary fails. I
> will try to share what i am able to do so far.
>
> But if anyone has a working example for this then it would be great.
>
> With best regards,
> Ashish
>
> On Tue, Apr 16, 2019, 9:37 PM Anthony Baker <ab...@pivotal.io> wrote:
>
>> The pattern I’ve seen used looks like this:
>>
>> User application (e.g. browser) >> Global load balancer >> Service
>> instances (e.g. tomcat) >> Geode cluster
>>
>> If you have the Geode clusters connected via WAN, you can redirect
>> traffic to different data centers by tweaking the LB config.
>>
>>
>> Anthony
>>
>>
>> On Apr 16, 2019, at 2:58 AM, aashish choudhary <
>> aashish.choudhary1@gmail.com> wrote:
>>
>> So with WAN we will be active/active all time. I agree hardest is to
>> figure out when data centers are actually down. We are evaluating multiple
>> approaches as of now.
>>
>> On that note would you recommend(possibility any since connection is
>> mostly tcp/ip) using some load balancer NGINX or something to handle data
>> center failure.
>>
>> With best regards,
>> Ashish
>>
>> On Tue, Apr 16, 2019, 12:54 AM Michael Stolz <ms...@pivotal.io> wrote:
>>
>>> We have come across these kinds of use-cases.
>>> The hardest part is figuring out that one of the data centers is
>>> ACTUALLY down.
>>>
>>> If you can work out a way to be active/active at all times and guard
>>> against update collisions by using data structures that protect themselves
>>> (e.g. CRDTs) that would make the whole thing a lot easier.
>>> --
>>> Mike Stolz
>>> Principal Engineer, GemFire Product Lead
>>> Mobile: +1-631-835-4771
>>>
>>>
>>>
>>> On Mon, Apr 15, 2019 at 1:19 PM aashish choudhary <
>>> aashish.choudhary1@gmail.com> wrote:
>>>
>>>> Thanks Mike. Our use case is heavily reliant on Geode(no fallback to
>>>> database) and business expectation is that there will be no downtime to
>>>> consumer application because of complete failure on one data center. Which
>>>>
>>>> Have you came across such cases with Geode/Gemfire?
>>>>
>>>> Regarding catching those exceptions and making a switch I agree with
>>>> you that it would be tricky to make switch as you explained.
>>>>
>>>> Even with rolling restart there will be a downtime and some manual
>>>> steps will be required to accomplish that.
>>>>
>>>> With best regards,
>>>> Ashish
>>>>
>>>> On Thu, Apr 11, 2019, 10:18 PM Michael Stolz <ms...@pivotal.io> wrote:
>>>>
>>>>> Yes you can catch the exceptions for no locators available and no
>>>>> servers available.
>>>>> You will probably want to wait for a period of time after first seeing
>>>>> this, because the cluster might be restarting and will be back in just a
>>>>> minute or so.
>>>>>
>>>>> The switch-over can be tricky without just restarting your client.
>>>>>
>>>>> All saved references to everything having to do with ClientCache,
>>>>> Cache, Region, or anything else that communicates need to be forgotten and
>>>>> re-established.
>>>>> This can be particularly challenging if you are using a framework that
>>>>> might remember some of this stuff on your behalf.
>>>>> I have usually recommended rolling restart of the clients with the new
>>>>> locator addresses because it is sure to work and not have any hidden issues
>>>>> with calls in progress or subscriptions or anything like that.
>>>>>
>>>>>
>>>>> --
>>>>> Mike Stolz
>>>>> Principal Engineer, GemFire Product Lead
>>>>> Mobile: +1-631-835-4771
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Apr 11, 2019 at 9:31 AM aashish choudhary <
>>>>> aashish.choudhary1@gmail.com> wrote:
>>>>>
>>>>>> Thanks Mike.
>>>>>>
>>>>>> Yes we are using wan replication. We want the switch to be an
>>>>>> automatic step. As soon as prod cluster fails we need to switch to cob
>>>>>> without any restart of the client application.
>>>>>>
>>>>>> One way we are thinking of is probably catching those locator not
>>>>>> available sort of exception and then make a switch. Any thoughts?
>>>>>>
>>>>>>
>>>>>> With best regards,
>>>>>> Ashish
>>>>>>
>>>>>>
>>>>>> On Thu, Apr 11, 2019, 1:42 AM Michael Stolz <ms...@pivotal.io>
>>>>>> wrote:
>>>>>>
>>>>>>> If the data centers are far apart you will want to use the
>>>>>>> bi-directional GemFire WAN Gateway to replicate between clusters.
>>>>>>>
>>>>>>> The trickiest part is figuring out when to switch. If you already
>>>>>>> have a mechanism for that then that's great.
>>>>>>>
>>>>>>> Once you know for sure you want to switch, the easiest way is to
>>>>>>> install a gemfire.properties file on the client machines that points to the
>>>>>>> locators in the other data center and restart the clients.
>>>>>>>
>>>>>>> There is a programmatic way to do it but is a lot more code and work
>>>>>>> than this way.
>>>>>>>
>>>>>>> Feel free to ask any additional questions here.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Mike Stolz
>>>>>>> Principal Engineer, GemFire Product Lead
>>>>>>> Mobile: +1-631-835-4771
>>>>>>>
>>>>>>> On Wed, Apr 10, 2019, 2:01 PM aashish choudhary <
>>>>>>> aashish.choudhary1@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> We have a scenario where in we need to switch over to a different
>>>>>>>> data center automatically when any failure occurs in the existing cluster.
>>>>>>>>
>>>>>>>> Any recommendations?
>>>>>>>>
>>>>>>>>
>>>>>>>> With best regards,
>>>>>>>> Ashish
>>>>>>>>
>>>>>>>
>>
Re: Geode switch over to different data center
Posted by aashish choudhary <aa...@gmail.com>.
Replying to old thread.
Currently I am trying test client side failover programtically. Are there
any working examples /best practices for this?
Idea is to failover to different secondary cluster if the primary fails. I
will try to share what i am able to do so far.
But if anyone has a working example for this then it would be great.
With best regards,
Ashish
On Tue, Apr 16, 2019, 9:37 PM Anthony Baker <ab...@pivotal.io> wrote:
> The pattern I’ve seen used looks like this:
>
> User application (e.g. browser) >> Global load balancer >> Service
> instances (e.g. tomcat) >> Geode cluster
>
> If you have the Geode clusters connected via WAN, you can redirect traffic
> to different data centers by tweaking the LB config.
>
>
> Anthony
>
>
> On Apr 16, 2019, at 2:58 AM, aashish choudhary <
> aashish.choudhary1@gmail.com> wrote:
>
> So with WAN we will be active/active all time. I agree hardest is to
> figure out when data centers are actually down. We are evaluating multiple
> approaches as of now.
>
> On that note would you recommend(possibility any since connection is
> mostly tcp/ip) using some load balancer NGINX or something to handle data
> center failure.
>
> With best regards,
> Ashish
>
> On Tue, Apr 16, 2019, 12:54 AM Michael Stolz <ms...@pivotal.io> wrote:
>
>> We have come across these kinds of use-cases.
>> The hardest part is figuring out that one of the data centers is ACTUALLY
>> down.
>>
>> If you can work out a way to be active/active at all times and guard
>> against update collisions by using data structures that protect themselves
>> (e.g. CRDTs) that would make the whole thing a lot easier.
>> --
>> Mike Stolz
>> Principal Engineer, GemFire Product Lead
>> Mobile: +1-631-835-4771
>>
>>
>>
>> On Mon, Apr 15, 2019 at 1:19 PM aashish choudhary <
>> aashish.choudhary1@gmail.com> wrote:
>>
>>> Thanks Mike. Our use case is heavily reliant on Geode(no fallback to
>>> database) and business expectation is that there will be no downtime to
>>> consumer application because of complete failure on one data center. Which
>>>
>>> Have you came across such cases with Geode/Gemfire?
>>>
>>> Regarding catching those exceptions and making a switch I agree with you
>>> that it would be tricky to make switch as you explained.
>>>
>>> Even with rolling restart there will be a downtime and some manual steps
>>> will be required to accomplish that.
>>>
>>> With best regards,
>>> Ashish
>>>
>>> On Thu, Apr 11, 2019, 10:18 PM Michael Stolz <ms...@pivotal.io> wrote:
>>>
>>>> Yes you can catch the exceptions for no locators available and no
>>>> servers available.
>>>> You will probably want to wait for a period of time after first seeing
>>>> this, because the cluster might be restarting and will be back in just a
>>>> minute or so.
>>>>
>>>> The switch-over can be tricky without just restarting your client.
>>>>
>>>> All saved references to everything having to do with ClientCache,
>>>> Cache, Region, or anything else that communicates need to be forgotten and
>>>> re-established.
>>>> This can be particularly challenging if you are using a framework that
>>>> might remember some of this stuff on your behalf.
>>>> I have usually recommended rolling restart of the clients with the new
>>>> locator addresses because it is sure to work and not have any hidden issues
>>>> with calls in progress or subscriptions or anything like that.
>>>>
>>>>
>>>> --
>>>> Mike Stolz
>>>> Principal Engineer, GemFire Product Lead
>>>> Mobile: +1-631-835-4771
>>>>
>>>>
>>>>
>>>> On Thu, Apr 11, 2019 at 9:31 AM aashish choudhary <
>>>> aashish.choudhary1@gmail.com> wrote:
>>>>
>>>>> Thanks Mike.
>>>>>
>>>>> Yes we are using wan replication. We want the switch to be an
>>>>> automatic step. As soon as prod cluster fails we need to switch to cob
>>>>> without any restart of the client application.
>>>>>
>>>>> One way we are thinking of is probably catching those locator not
>>>>> available sort of exception and then make a switch. Any thoughts?
>>>>>
>>>>>
>>>>> With best regards,
>>>>> Ashish
>>>>>
>>>>>
>>>>> On Thu, Apr 11, 2019, 1:42 AM Michael Stolz <ms...@pivotal.io> wrote:
>>>>>
>>>>>> If the data centers are far apart you will want to use the
>>>>>> bi-directional GemFire WAN Gateway to replicate between clusters.
>>>>>>
>>>>>> The trickiest part is figuring out when to switch. If you already
>>>>>> have a mechanism for that then that's great.
>>>>>>
>>>>>> Once you know for sure you want to switch, the easiest way is to
>>>>>> install a gemfire.properties file on the client machines that points to the
>>>>>> locators in the other data center and restart the clients.
>>>>>>
>>>>>> There is a programmatic way to do it but is a lot more code and work
>>>>>> than this way.
>>>>>>
>>>>>> Feel free to ask any additional questions here.
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Mike Stolz
>>>>>> Principal Engineer, GemFire Product Lead
>>>>>> Mobile: +1-631-835-4771
>>>>>>
>>>>>> On Wed, Apr 10, 2019, 2:01 PM aashish choudhary <
>>>>>> aashish.choudhary1@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> We have a scenario where in we need to switch over to a different
>>>>>>> data center automatically when any failure occurs in the existing cluster.
>>>>>>>
>>>>>>> Any recommendations?
>>>>>>>
>>>>>>>
>>>>>>> With best regards,
>>>>>>> Ashish
>>>>>>>
>>>>>>
>
Re: Geode switch over to different data center
Posted by Anthony Baker <ab...@pivotal.io>.
The pattern I’ve seen used looks like this:
User application (e.g. browser) >> Global load balancer >> Service instances (e.g. tomcat) >> Geode cluster
If you have the Geode clusters connected via WAN, you can redirect traffic to different data centers by tweaking the LB config.
Anthony
> On Apr 16, 2019, at 2:58 AM, aashish choudhary <aa...@gmail.com> wrote:
>
> So with WAN we will be active/active all time. I agree hardest is to figure out when data centers are actually down. We are evaluating multiple approaches as of now.
>
> On that note would you recommend(possibility any since connection is mostly tcp/ip) using some load balancer NGINX or something to handle data center failure.
>
> With best regards,
> Ashish
>
> On Tue, Apr 16, 2019, 12:54 AM Michael Stolz <mstolz@pivotal.io <ma...@pivotal.io>> wrote:
> We have come across these kinds of use-cases.
> The hardest part is figuring out that one of the data centers is ACTUALLY down.
>
> If you can work out a way to be active/active at all times and guard against update collisions by using data structures that protect themselves (e.g. CRDTs) that would make the whole thing a lot easier.
> --
> Mike Stolz
> Principal Engineer, GemFire Product Lead
> Mobile: +1-631-835-4771
>
>
>
> On Mon, Apr 15, 2019 at 1:19 PM aashish choudhary <aashish.choudhary1@gmail.com <ma...@gmail.com>> wrote:
> Thanks Mike. Our use case is heavily reliant on Geode(no fallback to database) and business expectation is that there will be no downtime to consumer application because of complete failure on one data center. Which
>
> Have you came across such cases with Geode/Gemfire?
>
> Regarding catching those exceptions and making a switch I agree with you that it would be tricky to make switch as you explained.
>
> Even with rolling restart there will be a downtime and some manual steps will be required to accomplish that.
>
> With best regards,
> Ashish
>
> On Thu, Apr 11, 2019, 10:18 PM Michael Stolz <mstolz@pivotal.io <ma...@pivotal.io>> wrote:
> Yes you can catch the exceptions for no locators available and no servers available.
> You will probably want to wait for a period of time after first seeing this, because the cluster might be restarting and will be back in just a minute or so.
>
> The switch-over can be tricky without just restarting your client.
>
> All saved references to everything having to do with ClientCache, Cache, Region, or anything else that communicates need to be forgotten and re-established.
> This can be particularly challenging if you are using a framework that might remember some of this stuff on your behalf.
> I have usually recommended rolling restart of the clients with the new locator addresses because it is sure to work and not have any hidden issues with calls in progress or subscriptions or anything like that.
>
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Lead
> Mobile: +1-631-835-4771
>
>
>
> On Thu, Apr 11, 2019 at 9:31 AM aashish choudhary <aashish.choudhary1@gmail.com <ma...@gmail.com>> wrote:
> Thanks Mike.
>
> Yes we are using wan replication. We want the switch to be an automatic step. As soon as prod cluster fails we need to switch to cob without any restart of the client application.
>
> One way we are thinking of is probably catching those locator not available sort of exception and then make a switch. Any thoughts?
>
>
> With best regards,
> Ashish
>
>
> On Thu, Apr 11, 2019, 1:42 AM Michael Stolz <mstolz@pivotal.io <ma...@pivotal.io>> wrote:
> If the data centers are far apart you will want to use the bi-directional GemFire WAN Gateway to replicate between clusters.
>
> The trickiest part is figuring out when to switch. If you already have a mechanism for that then that's great.
>
> Once you know for sure you want to switch, the easiest way is to install a gemfire.properties file on the client machines that points to the locators in the other data center and restart the clients.
>
> There is a programmatic way to do it but is a lot more code and work than this way.
>
> Feel free to ask any additional questions here.
>
>
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Lead
> Mobile: +1-631-835-4771
>
> On Wed, Apr 10, 2019, 2:01 PM aashish choudhary <aashish.choudhary1@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> We have a scenario where in we need to switch over to a different data center automatically when any failure occurs in the existing cluster.
>
> Any recommendations?
>
>
> With best regards,
> Ashish
Re: Geode switch over to different data center
Posted by aashish choudhary <aa...@gmail.com>.
So with WAN we will be active/active all time. I agree hardest is to figure
out when data centers are actually down. We are evaluating multiple
approaches as of now.
On that note would you recommend(possibility any since connection is mostly
tcp/ip) using some load balancer NGINX or something to handle data center
failure.
With best regards,
Ashish
On Tue, Apr 16, 2019, 12:54 AM Michael Stolz <ms...@pivotal.io> wrote:
> We have come across these kinds of use-cases.
> The hardest part is figuring out that one of the data centers is ACTUALLY
> down.
>
> If you can work out a way to be active/active at all times and guard
> against update collisions by using data structures that protect themselves
> (e.g. CRDTs) that would make the whole thing a lot easier.
> --
> Mike Stolz
> Principal Engineer, GemFire Product Lead
> Mobile: +1-631-835-4771
>
>
>
> On Mon, Apr 15, 2019 at 1:19 PM aashish choudhary <
> aashish.choudhary1@gmail.com> wrote:
>
>> Thanks Mike. Our use case is heavily reliant on Geode(no fallback to
>> database) and business expectation is that there will be no downtime to
>> consumer application because of complete failure on one data center. Which
>>
>> Have you came across such cases with Geode/Gemfire?
>>
>> Regarding catching those exceptions and making a switch I agree with you
>> that it would be tricky to make switch as you explained.
>>
>> Even with rolling restart there will be a downtime and some manual steps
>> will be required to accomplish that.
>>
>> With best regards,
>> Ashish
>>
>> On Thu, Apr 11, 2019, 10:18 PM Michael Stolz <ms...@pivotal.io> wrote:
>>
>>> Yes you can catch the exceptions for no locators available and no
>>> servers available.
>>> You will probably want to wait for a period of time after first seeing
>>> this, because the cluster might be restarting and will be back in just a
>>> minute or so.
>>>
>>> The switch-over can be tricky without just restarting your client.
>>>
>>> All saved references to everything having to do with ClientCache, Cache,
>>> Region, or anything else that communicates need to be forgotten and
>>> re-established.
>>> This can be particularly challenging if you are using a framework that
>>> might remember some of this stuff on your behalf.
>>> I have usually recommended rolling restart of the clients with the new
>>> locator addresses because it is sure to work and not have any hidden issues
>>> with calls in progress or subscriptions or anything like that.
>>>
>>>
>>> --
>>> Mike Stolz
>>> Principal Engineer, GemFire Product Lead
>>> Mobile: +1-631-835-4771
>>>
>>>
>>>
>>> On Thu, Apr 11, 2019 at 9:31 AM aashish choudhary <
>>> aashish.choudhary1@gmail.com> wrote:
>>>
>>>> Thanks Mike.
>>>>
>>>> Yes we are using wan replication. We want the switch to be an automatic
>>>> step. As soon as prod cluster fails we need to switch to cob without any
>>>> restart of the client application.
>>>>
>>>> One way we are thinking of is probably catching those locator not
>>>> available sort of exception and then make a switch. Any thoughts?
>>>>
>>>>
>>>> With best regards,
>>>> Ashish
>>>>
>>>>
>>>> On Thu, Apr 11, 2019, 1:42 AM Michael Stolz <ms...@pivotal.io> wrote:
>>>>
>>>>> If the data centers are far apart you will want to use the
>>>>> bi-directional GemFire WAN Gateway to replicate between clusters.
>>>>>
>>>>> The trickiest part is figuring out when to switch. If you already have
>>>>> a mechanism for that then that's great.
>>>>>
>>>>> Once you know for sure you want to switch, the easiest way is to
>>>>> install a gemfire.properties file on the client machines that points to the
>>>>> locators in the other data center and restart the clients.
>>>>>
>>>>> There is a programmatic way to do it but is a lot more code and work
>>>>> than this way.
>>>>>
>>>>> Feel free to ask any additional questions here.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mike Stolz
>>>>> Principal Engineer, GemFire Product Lead
>>>>> Mobile: +1-631-835-4771
>>>>>
>>>>> On Wed, Apr 10, 2019, 2:01 PM aashish choudhary <
>>>>> aashish.choudhary1@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We have a scenario where in we need to switch over to a different
>>>>>> data center automatically when any failure occurs in the existing cluster.
>>>>>>
>>>>>> Any recommendations?
>>>>>>
>>>>>>
>>>>>> With best regards,
>>>>>> Ashish
>>>>>>
>>>>>
Re: Geode switch over to different data center
Posted by Michael Stolz <ms...@pivotal.io>.
We have come across these kinds of use-cases.
The hardest part is figuring out that one of the data centers is ACTUALLY
down.
If you can work out a way to be active/active at all times and guard
against update collisions by using data structures that protect themselves
(e.g. CRDTs) that would make the whole thing a lot easier.
--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
On Mon, Apr 15, 2019 at 1:19 PM aashish choudhary <
aashish.choudhary1@gmail.com> wrote:
> Thanks Mike. Our use case is heavily reliant on Geode(no fallback to
> database) and business expectation is that there will be no downtime to
> consumer application because of complete failure on one data center. Which
>
> Have you came across such cases with Geode/Gemfire?
>
> Regarding catching those exceptions and making a switch I agree with you
> that it would be tricky to make switch as you explained.
>
> Even with rolling restart there will be a downtime and some manual steps
> will be required to accomplish that.
>
> With best regards,
> Ashish
>
> On Thu, Apr 11, 2019, 10:18 PM Michael Stolz <ms...@pivotal.io> wrote:
>
>> Yes you can catch the exceptions for no locators available and no servers
>> available.
>> You will probably want to wait for a period of time after first seeing
>> this, because the cluster might be restarting and will be back in just a
>> minute or so.
>>
>> The switch-over can be tricky without just restarting your client.
>>
>> All saved references to everything having to do with ClientCache, Cache,
>> Region, or anything else that communicates need to be forgotten and
>> re-established.
>> This can be particularly challenging if you are using a framework that
>> might remember some of this stuff on your behalf.
>> I have usually recommended rolling restart of the clients with the new
>> locator addresses because it is sure to work and not have any hidden issues
>> with calls in progress or subscriptions or anything like that.
>>
>>
>> --
>> Mike Stolz
>> Principal Engineer, GemFire Product Lead
>> Mobile: +1-631-835-4771
>>
>>
>>
>> On Thu, Apr 11, 2019 at 9:31 AM aashish choudhary <
>> aashish.choudhary1@gmail.com> wrote:
>>
>>> Thanks Mike.
>>>
>>> Yes we are using wan replication. We want the switch to be an automatic
>>> step. As soon as prod cluster fails we need to switch to cob without any
>>> restart of the client application.
>>>
>>> One way we are thinking of is probably catching those locator not
>>> available sort of exception and then make a switch. Any thoughts?
>>>
>>>
>>> With best regards,
>>> Ashish
>>>
>>>
>>> On Thu, Apr 11, 2019, 1:42 AM Michael Stolz <ms...@pivotal.io> wrote:
>>>
>>>> If the data centers are far apart you will want to use the
>>>> bi-directional GemFire WAN Gateway to replicate between clusters.
>>>>
>>>> The trickiest part is figuring out when to switch. If you already have
>>>> a mechanism for that then that's great.
>>>>
>>>> Once you know for sure you want to switch, the easiest way is to
>>>> install a gemfire.properties file on the client machines that points to the
>>>> locators in the other data center and restart the clients.
>>>>
>>>> There is a programmatic way to do it but is a lot more code and work
>>>> than this way.
>>>>
>>>> Feel free to ask any additional questions here.
>>>>
>>>>
>>>>
>>>> --
>>>> Mike Stolz
>>>> Principal Engineer, GemFire Product Lead
>>>> Mobile: +1-631-835-4771
>>>>
>>>> On Wed, Apr 10, 2019, 2:01 PM aashish choudhary <
>>>> aashish.choudhary1@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> We have a scenario where in we need to switch over to a different data
>>>>> center automatically when any failure occurs in the existing cluster.
>>>>>
>>>>> Any recommendations?
>>>>>
>>>>>
>>>>> With best regards,
>>>>> Ashish
>>>>>
>>>>
Re: Geode switch over to different data center
Posted by aashish choudhary <aa...@gmail.com>.
Thanks Mike. Our use case is heavily reliant on Geode(no fallback to
database) and business expectation is that there will be no downtime to
consumer application because of complete failure on one data center. Which
Have you came across such cases with Geode/Gemfire?
Regarding catching those exceptions and making a switch I agree with you
that it would be tricky to make switch as you explained.
Even with rolling restart there will be a downtime and some manual steps
will be required to accomplish that.
With best regards,
Ashish
On Thu, Apr 11, 2019, 10:18 PM Michael Stolz <ms...@pivotal.io> wrote:
> Yes you can catch the exceptions for no locators available and no servers
> available.
> You will probably want to wait for a period of time after first seeing
> this, because the cluster might be restarting and will be back in just a
> minute or so.
>
> The switch-over can be tricky without just restarting your client.
>
> All saved references to everything having to do with ClientCache, Cache,
> Region, or anything else that communicates need to be forgotten and
> re-established.
> This can be particularly challenging if you are using a framework that
> might remember some of this stuff on your behalf.
> I have usually recommended rolling restart of the clients with the new
> locator addresses because it is sure to work and not have any hidden issues
> with calls in progress or subscriptions or anything like that.
>
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Lead
> Mobile: +1-631-835-4771
>
>
>
> On Thu, Apr 11, 2019 at 9:31 AM aashish choudhary <
> aashish.choudhary1@gmail.com> wrote:
>
>> Thanks Mike.
>>
>> Yes we are using wan replication. We want the switch to be an automatic
>> step. As soon as prod cluster fails we need to switch to cob without any
>> restart of the client application.
>>
>> One way we are thinking of is probably catching those locator not
>> available sort of exception and then make a switch. Any thoughts?
>>
>>
>> With best regards,
>> Ashish
>>
>>
>> On Thu, Apr 11, 2019, 1:42 AM Michael Stolz <ms...@pivotal.io> wrote:
>>
>>> If the data centers are far apart you will want to use the
>>> bi-directional GemFire WAN Gateway to replicate between clusters.
>>>
>>> The trickiest part is figuring out when to switch. If you already have a
>>> mechanism for that then that's great.
>>>
>>> Once you know for sure you want to switch, the easiest way is to install
>>> a gemfire.properties file on the client machines that points to the
>>> locators in the other data center and restart the clients.
>>>
>>> There is a programmatic way to do it but is a lot more code and work
>>> than this way.
>>>
>>> Feel free to ask any additional questions here.
>>>
>>>
>>>
>>> --
>>> Mike Stolz
>>> Principal Engineer, GemFire Product Lead
>>> Mobile: +1-631-835-4771
>>>
>>> On Wed, Apr 10, 2019, 2:01 PM aashish choudhary <
>>> aashish.choudhary1@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We have a scenario where in we need to switch over to a different data
>>>> center automatically when any failure occurs in the existing cluster.
>>>>
>>>> Any recommendations?
>>>>
>>>>
>>>> With best regards,
>>>> Ashish
>>>>
>>>
Re: Geode switch over to different data center
Posted by Michael Stolz <ms...@pivotal.io>.
Yes you can catch the exceptions for no locators available and no servers
available.
You will probably want to wait for a period of time after first seeing
this, because the cluster might be restarting and will be back in just a
minute or so.
The switch-over can be tricky without just restarting your client.
All saved references to everything having to do with ClientCache, Cache,
Region, or anything else that communicates need to be forgotten and
re-established.
This can be particularly challenging if you are using a framework that
might remember some of this stuff on your behalf.
I have usually recommended rolling restart of the clients with the new
locator addresses because it is sure to work and not have any hidden issues
with calls in progress or subscriptions or anything like that.
--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
On Thu, Apr 11, 2019 at 9:31 AM aashish choudhary <
aashish.choudhary1@gmail.com> wrote:
> Thanks Mike.
>
> Yes we are using wan replication. We want the switch to be an automatic
> step. As soon as prod cluster fails we need to switch to cob without any
> restart of the client application.
>
> One way we are thinking of is probably catching those locator not
> available sort of exception and then make a switch. Any thoughts?
>
>
> With best regards,
> Ashish
>
>
> On Thu, Apr 11, 2019, 1:42 AM Michael Stolz <ms...@pivotal.io> wrote:
>
>> If the data centers are far apart you will want to use the bi-directional
>> GemFire WAN Gateway to replicate between clusters.
>>
>> The trickiest part is figuring out when to switch. If you already have a
>> mechanism for that then that's great.
>>
>> Once you know for sure you want to switch, the easiest way is to install
>> a gemfire.properties file on the client machines that points to the
>> locators in the other data center and restart the clients.
>>
>> There is a programmatic way to do it but is a lot more code and work than
>> this way.
>>
>> Feel free to ask any additional questions here.
>>
>>
>>
>> --
>> Mike Stolz
>> Principal Engineer, GemFire Product Lead
>> Mobile: +1-631-835-4771
>>
>> On Wed, Apr 10, 2019, 2:01 PM aashish choudhary <
>> aashish.choudhary1@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> We have a scenario where in we need to switch over to a different data
>>> center automatically when any failure occurs in the existing cluster.
>>>
>>> Any recommendations?
>>>
>>>
>>> With best regards,
>>> Ashish
>>>
>>
Re: Geode switch over to different data center
Posted by aashish choudhary <aa...@gmail.com>.
Thanks Mike.
Yes we are using wan replication. We want the switch to be an automatic
step. As soon as prod cluster fails we need to switch to cob without any
restart of the client application.
One way we are thinking of is probably catching those locator not available
sort of exception and then make a switch. Any thoughts?
With best regards,
Ashish
On Thu, Apr 11, 2019, 1:42 AM Michael Stolz <ms...@pivotal.io> wrote:
> If the data centers are far apart you will want to use the bi-directional
> GemFire WAN Gateway to replicate between clusters.
>
> The trickiest part is figuring out when to switch. If you already have a
> mechanism for that then that's great.
>
> Once you know for sure you want to switch, the easiest way is to install a
> gemfire.properties file on the client machines that points to the locators
> in the other data center and restart the clients.
>
> There is a programmatic way to do it but is a lot more code and work than
> this way.
>
> Feel free to ask any additional questions here.
>
>
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Lead
> Mobile: +1-631-835-4771
>
> On Wed, Apr 10, 2019, 2:01 PM aashish choudhary <
> aashish.choudhary1@gmail.com> wrote:
>
>> Hi,
>>
>> We have a scenario where in we need to switch over to a different data
>> center automatically when any failure occurs in the existing cluster.
>>
>> Any recommendations?
>>
>>
>> With best regards,
>> Ashish
>>
>
Re: Geode switch over to different data center
Posted by Michael Stolz <ms...@pivotal.io>.
If the data centers are far apart you will want to use the bi-directional
GemFire WAN Gateway to replicate between clusters.
The trickiest part is figuring out when to switch. If you already have a
mechanism for that then that's great.
Once you know for sure you want to switch, the easiest way is to install a
gemfire.properties file on the client machines that points to the locators
in the other data center and restart the clients.
There is a programmatic way to do it but is a lot more code and work than
this way.
Feel free to ask any additional questions here.
--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
On Wed, Apr 10, 2019, 2:01 PM aashish choudhary <
aashish.choudhary1@gmail.com> wrote:
> Hi,
>
> We have a scenario where in we need to switch over to a different data
> center automatically when any failure occurs in the existing cluster.
>
> Any recommendations?
>
>
> With best regards,
> Ashish
>