ZK + dynamic config + EC2

["Oboturov, Artem" <ar...@zalando.de>]

Hi

I was looking at the ZK 3.5.x series new feature - dynamic configuration.
As an example, we could have an EC2 auto scaling group for 3 ZK nodes. When
one of them goes down, a new one would be spawned, but its IP could be
different. We could query EC2 to get all instances from group and generate
a config for ZK to take all currently running servers as part of ZK
cluster, and then run an update of cluster configs for all existing ones
using dynamic config feature. Would this strategy work? Are there any
alternatives?

-- 
Regards
Artem Oboturov

Re: ZK + dynamic config + EC2

[Alexander Shraer <sh...@gmail.com>]

Yes, unique ids require consensus (or admin), but ZK gives you this. you
could use for example the sequential flag or alternatively the set
conditional on a version.
On Nov 22, 2015 3:34 PM, "Oboturov, Artem" <ar...@zalando.de>
wrote:

> There is one more thing to it: ZK has a byte range for values server IDs -
> 255 in total, how could you allocate those to instances without having a
> central registry for IDs? It could be based on IPs, but it seems there are
> no other idempotent/reliable ways to get them assigned?
>
> On 21 November 2015 at 20:42, Alexander Shraer <sh...@gmail.com> wrote:
>
>> The only issue I see is that if the new server has the same id as the old
>> one you're replacing, I think you should first remove the old one and then
>> in a separate command add the new one. Intuitively this way you avoid
>> having the newly joining server act as someone who knows the current state
>> of the system (which it doesn't), a situation that may cause you to loose
>> transactions.
>>
>> Notice that there are two reconfig interfaces  -- incremental and bulk
>> (see manual
>> <https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html>).
>>
>> On Sat, Nov 21, 2015 at 1:16 PM, Oboturov, Artem <
>> artem.oboturov@zalando.de> wrote:
>>
>>> Hi
>>>
>>> I was looking at the ZK 3.5.x series new feature - dynamic configuration.
>>> As an example, we could have an EC2 auto scaling group for 3 ZK nodes.
>>> When
>>> one of them goes down, a new one would be spawned, but its IP could be
>>> different. We could query EC2 to get all instances from group and
>>> generate
>>> a config for ZK to take all currently running servers as part of ZK
>>> cluster, and then run an update of cluster configs for all existing ones
>>> using dynamic config feature. Would this strategy work? Are there any
>>> alternatives?
>>>
>>> --
>>> Regards
>>> Artem Oboturov
>>>
>>
>>
>
>
> --
> Regards
> Artem Oboturov
>

Re: ZK + dynamic config + EC2

[Alexander Shraer <sh...@gmail.com>]

The only issue I see is that if the new server has the same id as the old
one you're replacing, I think you should first remove the old one and then
in a separate command add the new one. Intuitively this way you avoid
having the newly joining server act as someone who knows the current state
of the system (which it doesn't), a situation that may cause you to loose
transactions.

Notice that there are two reconfig interfaces  -- incremental and bulk (see
manual <https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html>).

On Sat, Nov 21, 2015 at 1:16 PM, Oboturov, Artem <ar...@zalando.de>
wrote:

> Hi
>
> I was looking at the ZK 3.5.x series new feature - dynamic configuration.
> As an example, we could have an EC2 auto scaling group for 3 ZK nodes. When
> one of them goes down, a new one would be spawned, but its IP could be
> different. We could query EC2 to get all instances from group and generate
> a config for ZK to take all currently running servers as part of ZK
> cluster, and then run an update of cluster configs for all existing ones
> using dynamic config feature. Would this strategy work? Are there any
> alternatives?
>
> --
> Regards
> Artem Oboturov
>

Re: ZK + dynamic config + EC2

[Raúl Gutiérrez Segalés <rg...@itevenworks.net>]

Hi,

On 21 November 2015 at 10:16, Oboturov, Artem <ar...@zalando.de>
wrote:

> Hi
>
> I was looking at the ZK 3.5.x series new feature - dynamic configuration.
> As an example, we could have an EC2 auto scaling group for 3 ZK nodes. When
> one of them goes down, a new one would be spawned, but its IP could be
> different. We could query EC2 to get all instances from group and generate
> a config for ZK to take all currently running servers as part of ZK
> cluster, and then run an update of cluster configs for all existing ones
> using dynamic config feature. Would this strategy work? Are there any
> alternatives?
>

Yeah, that should work. You'd have to call reconfig() to remove the old
server
and add the new one.


-rgs