Dynamic locators property

[Nadine Daccache <an...@gmail.com>]

Hello everyone,



I have a problem related to the locators property used to start the members
of a cluster, I thought you might help.



In my case, the IP addresses of the locators hosts could not be statically
configured, and are only known at runtime. The idea is to register the
locator host and port in a central registry, and whenever a locator is
started, to check the previously started locators.



For instance:

-          locator1 will be started with no locators property, and
registers host1[port1]

-          locator2 will be started with --locators=host1[port1], and
registers host2[port2]

-          locator3 will be started with
--locators=host1[port1],host2[port2], and registers host3[port3]



I have tried it, and it seems to work fine. The 3 locators join the same
distributed system. But I haven’t performed more advanced testing.



The documentation clearly states that every locator should have the
complete locators list, even if they have not been started.



Do you know what would be the limitations (if any) if the locators are
started in this way? Do you have any idea that could help?



Regards,

Nadine

Re: Dynamic locators property

[Nadine Daccache <an...@gmail.com>]

Hello Udo,



Thank you for your detailed reply and useful hints!



The startup procedure will enforce the locators to start before the servers
and the clients. Therefore, both servers and clients will have the complete
locators list populated from the registry.



I understand that this is not a standard/recommended way, but we have some
technical constraints. To give you more details on the use case, we are
using Geode as third-party library in a platform that already has its own
middleware layer and its central registry. It’s not running in a
virtualized environment, but we do not want to introduce additional
deployment and configuration complexity for our clients.



The registry monitors the status of the registered processes, and it is
possible to detect locator failures. If a locator fails, it will be
restarted on the same host and port. Therefore, there shouldn’t be a “new
locator” issue.



WAN replication is not a requirement in our case. But I didn’t check how
the split brain would work in this case, I will give it a try and keep you
posted.



Best regards,

Nadine

2016-06-22 21:18 GMT+02:00 Udo Kohlmeyer <uk...@pivotal.io>:

> Hi there Nadine,
> This is an interesting scenario.... As you have already mentioned, the
> locators property is recommended to be static and defined up front.
> From what I understand the list of locators is build up dynamically at
> runtime. i.e there must be some process that runs before the locators
> property is set, that checks if there is an existing locator running and
> populates its credentials into the property.
>
> If I "walk through" the start up process you have defined, it makes sense
> that it could work. (so please don't take this as a blessing) It seems that
> it would work for locators... but I'm not sure about what it means for the
> servers or clients.
> Also are you intending to us WAN replication? If so, how is that handled?
> As the locators need a well-defined list of "remote-locators" at start up.
>
> It also brings up a whole bunch of other questions for servers, clients
> and HA.
>
> How does the central registry handle locators failing/stopping?
>
> How does it handle the case if locator2 were to fail? I mean, locator3
> would still validly reference it, but it might not exist any more. (which
> is fine). But if you now start locator4 (due to HA procedures), neither
> locator1 or locator3 would know about locator4. (or locator5 or locator6)
>
> Does the central registry monitor the locators to confirm that they still
> work and if not does it removes them from the list? What about connectivity
> issues (split brain, etc...)
>
> Is the start up procedure for the clusters ALWAYS the same? i.e all
> locators are started before the servers are started. Providing the "full"
> list of locators to the servers at startup.
>
> How do servers handle the failure of locators? What happens if all
> locators were to fail? The servers would still work, but reference
> incorrect locator host names. So the servers would not know about any of
> the "new" locators joining.
>
> How do clients get the full list of locators?
> How are the pools defined? i.e do clients follow the same process? Check
> the central registry for running locators and populate the pool locators
> list with only the registered locators?
>
> What happens when clients start before all the locators have been started?
> The ability for clients to connect/reconnect to clusters in case of failure
> is through the defined list of locators/servers. If it detects failure in
> the one, it will try the next in the list.... transparent failover.
>
> How does a running client handle the failure of all the locators? In case
> of all locators failing, does it mean that all servers and clients have to
> be shutdown and the whole eco-system has to be restarted?
>
> So maybe this approach is not advisable.
>
> If I may ask, why would you not be able to determine a set a servers that
> are used for the cluster? Could you not somehow reserve some hosts for the
> Geode cluster, without dynamic name allocation? Are you running in a
> virtualized environment like AWS, VMware?
>
> --Udo
>
>
> On 22/06/2016 10:24 PM, Nadine Daccache wrote:
>
> Hello everyone,
>
>
>
> I have a problem related to the locators property used to start the
> members of a cluster, I thought you might help.
>
>
>
> In my case, the IP addresses of the locators hosts could not be statically
> configured, and are only known at runtime. The idea is to register the
> locator host and port in a central registry, and whenever a locator is
> started, to check the previously started locators.
>
>
>
> For instance:
>
> -          locator1 will be started with no locators property, and
> registers host1[port1]
>
> -          locator2 will be started with --locators=host1[port1], and
> registers host2[port2]
>
> -          locator3 will be started with
> --locators=host1[port1],host2[port2], and registers host3[port3]
>
>
>
> I have tried it, and it seems to work fine. The 3 locators join the same
> distributed system. But I haven’t performed more advanced testing.
>
>
>
> The documentation clearly states that every locator should have the
> complete locators list, even if they have not been started.
>
>
>
> Do you know what would be the limitations (if any) if the locators are
> started in this way? Do you have any idea that could help?
>
>
>
> Regards,
>
> Nadine
>
>
>

Re: Dynamic locators property

[Udo Kohlmeyer <uk...@pivotal.io>]

Hi there Nadine,

This is an interesting scenario.... As you have already mentioned, the 
locators property is recommended to be static and defined up front.
 From what I understand the list of locators is build up dynamically at 
runtime. i.e there must be some process that runs before the locators 
property is set, that checks if there is an existing locator running and 
populates its credentials into the property.

If I "walk through" the start up process you have defined, it makes 
sense that it could work. (so please don't take this as a blessing) It 
seems that it would work for locators... but I'm not sure about what it 
means for the servers or clients.
Also are you intending to us WAN replication? If so, how is that 
handled? As the locators need a well-defined list of "remote-locators" 
at start up.

It also brings up a whole bunch of other questions for servers, clients 
and HA.

How does the central registry handle locators failing/stopping?

How does it handle the case if locator2 were to fail? I mean, locator3 
would still validly reference it, but it might not exist any more. 
(which is fine). But if you now start locator4 (due to HA procedures), 
neither locator1 or locator3 would know about locator4. (or locator5 or 
locator6)

Does the central registry monitor the locators to confirm that they 
still work and if not does it removes them from the list? What about 
connectivity issues (split brain, etc...)

Is the start up procedure for the clusters ALWAYS the same? i.e all 
locators are started before the servers are started. Providing the 
"full" list of locators to the servers at startup.

How do servers handle the failure of locators? What happens if all 
locators were to fail? The servers would still work, but reference 
incorrect locator host names. So the servers would not know about any of 
the "new" locators joining.

How do clients get the full list of locators?
How are the pools defined? i.e do clients follow the same process? Check 
the central registry for running locators and populate the pool locators 
list with only the registered locators?

What happens when clients start before all the locators have been 
started? The ability for clients to connect/reconnect to clusters in 
case of failure is through the defined list of locators/servers. If it 
detects failure in the one, it will try the next in the list.... 
transparent failover.

How does a running client handle the failure of all the locators? In 
case of all locators failing, does it mean that all servers and clients 
have to be shutdown and the whole eco-system has to be restarted?

So maybe this approach is not advisable.

If I may ask, why would you not be able to determine a set a servers 
that are used for the cluster? Could you not somehow reserve some hosts 
for the Geode cluster, without dynamic name allocation? Are you running 
in a virtualized environment like AWS, VMware?

--Udo

On 22/06/2016 10:24 PM, Nadine Daccache wrote:
>
> Hello everyone,
>
> I have a problem related to the locators property used to start the 
> members of a cluster, I thought you might help.
>
> In my case, the IP addresses of the locators hosts could not be 
> statically configured, and are only known at runtime. The idea is to 
> register the locator host and port in a central registry, and whenever 
> a locator is started, to check the previously started locators.
>
> For instance:
>
> -locator1 will be started with no locators property, and registers 
> host1[port1]
>
> -locator2 will be started with --locators=host1[port1], and registers 
> host2[port2]
>
> -locator3 will be started with --locators=host1[port1],host2[port2], 
> and registers host3[port3]
>
> I have tried it, and it seems to work fine. The 3 locators join the 
> same distributed system. But I haven\u2019t performed more advanced testing.
>
> The documentation clearly states that every locator should have the 
> complete locators list, even if they have not been started.
>
> Do you know what would be the limitations (if any) if the locators are 
> started in this way? Do you have any idea that could help?
>
> Regards,
>
> Nadine
>