You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Benjamin Reed <br...@apache.org> on 2021/03/02 05:03:38 UTC
Re: Ephemeral Nodes
wow, sorry for the late reply. it looks like you are using sequence
znodes. for this simple case, it seems like you could just use a
single znode and whoever creates it wins and the others watch. if you
don't want S1 to be the leader, why do you take it into account when
figuring out who is the leader. it sounds like you want to match
prefixes when seeing who is the leader.
have you thought about using curator to implement this? i think they
have the functionality you want already implemented.
ben
On Fri, Feb 12, 2021 at 6:36 PM Dipti Mulay <di...@4skylabs.com> wrote:
>
> Hi Ben ,
>
> What you said is correct.
>
> Here is the situation in a little more detail:
> - My cluster is made of many nodes some which are simple nodes (Single VM) lets calls them S1, S2, S3 etc and some are duplex nodes D1, D2 , etc.( a pair of two VMs acting as active-standby)
> - Let's say the pair of two VMs in duplex system are named A and B. So we have D1-A: D1-B, D2-A :D2-B etc.
> - I start all nodes at the same time:
> - Both D1-A , D1-B try to create an ephemeral node D1
> - S1,S2,S3 create an ephemeral node S1,S2,S3 etc.
> - D2-A, D2-B try create an ephemeral node D2
>
> - Now let's say following nodes are created (notes all VMs were started at the same time,, so I cannot control in which order nodes get created)
> - D1_000000000
> - S1_000000001
> - S2_000000002
> -D1_000000003
> -D2_000000004
> -D2_000000005
> - Note that there are two nodes for D1 and D2 : that is because each of the servers D1-A, D1-B, D2-A and D2-B succeed in creating the nodes.
> - Clearly D1-A (which created node D1_00000000) becomes the master ... all is good till this time.
> - Now during normal course of operations.... something bad happens on the node D1-A and it goes away. D1-B being the standby takes over the responsibilities of D1-A
> - However, according to the order above S1 will become the next leader of the cluster.
> - But what I would like to happen is that D1-B becomes the leader of the lot. Is this possible in any way ?
>
>
> Hope I am making my self clear.
>
> Thanks
> -Parag
>
>
> On 2/13/21, 12:16 AM, "Benjamin Reed" <br...@apache.org> wrote:
>
> the simple leader election should work for you: both try to create an
> ephemeral node: /leader. the one that succeeds is the leader. sets a
> watch on /leader and will try to create it if it goes away. if the
> standby succeeds, it will become the leader and will remain so.
>
> (i realize i'm repeating what flavio said :) i though perhaps
> different words would help :) is there some functionality that this
> solution is missing? it is one of the most popular ways to do leader
> election with zookeeper.
>
> ben
>
> On Sat, Jan 30, 2021 at 1:35 AM Dipti Mulay <di...@4skylabs.com> wrote:
> >
> > Hi Flavio,
> >
> > I am using the Zookeepers to do a leader election. The leader election
> > needs to be done between entities which are actually a combination of
> > active/stanb-by machines. The standby is a hot standby ready to take over
> > when the active fails. Now when such a failure occurs I need to ensure that
> > if the active system was the leader then the standby system remains as
> > leader when it takes over and becomes active. Now I can have both active
> > and standby systems create their own ephemeral node. But that will not
> > guarantee that every time a failure happens on the active system the
> > standby which is going to become active will assume the role of leader;
> > that is because other ephemeral nodes might exist already.
> >
> > I am hoping that I explained it properly.
> >
> > Thanks for bearing with me.
> > -Parag
> >
> > On Fri, 29 Jan 2021 at 20:25, Flavio Junqueira <fp...@apache.org> wrote:
> >
> > > If the session expires, then you cannot reconnect to that session and the
> > > associated ephemerals should be deleted.
> > >
> > > I'm not entirely clear why you want to preserve the ephemeral znode. The
> > > session expiring and the ephemeral being deleted is an indication that the
> > > active leadership has been released, which brings to me to the question of
> > > how the standby is checking whether the active is still up. If it is not by
> > > watching the ephemeral znode, then it must be by other means.
> > >
> > > If you need multiple clients to know which client is active, then you
> > > might consider writing an id to a persistent znode. That alone won't be
> > > sufficient, though.
> > >
> > > -Flavio
> > >
> > > > On 29 Jan 2021, at 15:03, Dipti Mulay <di...@4skylabs.com> wrote:
> > > >
> > > > In his mail Ben said "when the session expires, the ephemeral nodes will
> > > > get deleted. if your standby system reconnects to zookeeper with the
> > > > clientid, the session will not expire and the ephemeral nodes will stay
> > > > active."
> > > > Are you suggesting that I can re-connect on a session that has already
> > > > expired..? Won't the eph node get deleted when the session expires?
> > > >
> > > > On Fri, 29 Jan 2021 at 18:33, Dipti Mulay <di...@4skylabs.com> wrote:
> > > >
> > > >> Hi Ben, Flavio,
> > > >>
> > > >> Thanks for your replies.
> > > >>
> > > >> I have tried coding it in a way that when standby takes over from the
> > > >> active system only then it will establish a session with ZK server using
> > > >> the same client id. But there is a small window of time when both
> > > systems
> > > >> have an active session.
> > > >> What I want to happen is that when standby takes over and active goes
> > > down
> > > >> the ephemeral node should remain intact. The reason being that if the
> > > >> active server was the "leader" then after the standby takes over I would
> > > >> like the standby to remain as leader.
> > > >> I have tried it many ways but as soon as the standby takes over and
> > > socket
> > > >> from the old active closes the ephemeral node is deleted and triggers
> > > >> leader election again.
> > > >>
> > > >> Hope this helps.
> > > >>
> > > >> Thanks
> > > >> -Parag
> > > >>
> > > >>
> > > >> On Fri, 29 Jan 2021 at 13:26, Flavio Junqueira <fp...@apache.org> wrote:
> > > >>
> > > >>> Yeah, what Ben said... What you might want to do is something along the
> > > >>> lines of:
> > > >>>
> > > >>> 1- Have a client create the ephemeral znode, let's call it say
> > > "/active"
> > > >>> 2- The other client acting as a standby sets a watch on "/active"
> > > >>> 3- If the active node crashes, then "/active" is removed and the
> > > standby
> > > >>> is notified and can become active by creating the "/active" znode
> > > >>>
> > > >>> Would that solve your problem?
> > > >>>
> > > >>> -Flavio
> > > >>>
> > > >>>> On 29 Jan 2021, at 07:44, Benjamin Reed <br...@apache.org> wrote:
> > > >>>>
> > > >>>> can you explain a bit more what you mean by "The zookeeper-client
> > > >>>> process on standby which uses the same clientid to initialize a
> > > >>>> session with zookeeper." ? if i understand correctly, you are going to
> > > >>>> have two clients using the same session which cannot happen. do you
> > > >>>> only reuse the clientid when the takeover occurs (the other node stops
> > > >>>> using the session)?
> > > >>>>
> > > >>>> when the session expires, the ephemeral nodes will get deleted. if
> > > >>>> your standby system reconnects to zookeeper with the clientid, the
> > > >>>> session will not expire and the ephemeral nodes will stay active.
> > > >>>>
> > > >>>> ben
> > > >>>>
> > > >>>> On Wed, Jan 27, 2021 at 3:20 AM Dipti Mulay <di...@4skylabs.com>
> > > wrote:
> > > >>>>>
> > > >>>>> Hi Flavio,
> > > >>>>>
> > > >>>>> Thanks for your inputs let me say what I am trying to do:
> > > >>>>>
> > > >>>>> I have an active- standby system, each of these system is running on
> > > >>> its own VM.
> > > >>>>>
> > > >>>>> The zookpeeper client process which communicates with ZK is running
> > > >>> even on the stanby system. The active system creates a session with
> > > ZK and
> > > >>> also creates a ephemeral node, I use the zoo_client_id() call to get
> > > the
> > > >>> clientId_t information.
> > > >>>>> The active system then passes on this ClientID_t to standby system(
> > > >>> this is done by a memory shadow mechanism we have). The
> > > zookeeper-client
> > > >>> process on standby which uses the same clientid to initialize a session
> > > >>> with zookeeper.
> > > >>>>>
> > > >>>>> Now after some time I turn of the active VM. On the stanby I see logs
> > > >>> which tell me a socket error happened and I get a session expired event
> > > >>> upon which I close the zookeeper session. Does the session expire even
> > > >>> also mean that my ephemeral is now gone? The standard error handling
> > > in my
> > > >>> code is that when a session expired event is received I call
> > > >>> zookeeper_close() which will surely remove the ephemeral node.
> > > >>>>> Is there a way to preserve the ephemeral node and re-connect the
> > > >>> session ?
> > > >>>>>
> > > >>>>> Thanks
> > > >>>>> -Parag
> > > >>>>> P.S: I hope I have clearly explained the problem
> > > >>>>>
> > > >>>>> On 1/26/21, 8:17 PM, "Flavio Junqueira" <fp...@apache.org> wrote:
> > > >>>>>
> > > >>>>> Hi Parag,
> > > >>>>>
> > > >>>>> Typically, the application knows the path of the znode that it
> > > >>> wants to check. Knowing the path, you can, for example, check the znode
> > > >>> metadata, which includes the session id indicating the owner in the
> > > case of
> > > >>> an ephemeral znode.
> > > >>>>>
> > > >>>>> -Flavio
> > > >>>>>
> > > >>>>>> On 26 Jan 2021, at 03:07, Dipti Mulay <di...@4skylabs.com> wrote:
> > > >>>>>>
> > > >>>>>> Hi All,
> > > >>>>>>
> > > >>>>>> I had a question regarding ephemeral nodes. Given a particular
> > > >>> client_id can zookeeper return me the ephemeral node corresponding to
> > > the
> > > >>> provided client_id?
> > > >>>>>> I am using the C Client Library and I could not find a way ?
> > > >>>>>>
> > > >>>>>> Any suggestions ?
> > > >>>>>>
> > > >>>>>> Thanks
> > > >>>>>> -Parag
> > > >>>>>
> > > >>>
> > > >>>
> > >
> > >