You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Ivan Kelly <iv...@apache.org> on 2013/03/13 22:01:40 UTC
What should I do with SyncDisconnected
Hi guys,
We have a usecase here where zookeeper is used to coordinate ownership
of partitions of a resource. When one server dies, the partition
should be moved to another server, etc. The action we need to take on
SessionExpired is very clear. We just kill the server.
However it is unclear what we should do on a SyncDisconnected. We
can't just kill our server, as it may have just been one zookeeper
server failing. If we block all client requests to our server while we
wait for SyncConnected, we may block forever in the case that our
server is partitioned away from the zk cluster. If we continue to
serve requests, we risk split brain[1].
What have people done in the past to resolve issues like this?
-Ivan
[1] This is a risk anyhow without proper fencing, but a limited amount
is ok in our application.
Re: What should I do with SyncDisconnected
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
Curator has its own concept of connection timeout. It's a separate value from the session timeout.
-JZ
On Mar 25, 2013, at 2:26 AM, Ivan Kelly <iv...@apache.org> wrote:
> On Thu, Mar 14, 2013 at 02:43:58PM -0700, Jordan Zimmerman wrote:
>>> But how long to wait? If the server is truly partitioned from ZK, then
>>> I'll wait forever, and the client request will be hung forever.
>> Curator waits for the time specified as the "connection
>> timeout". Curator introduces three connection states, CONNECTED,
>> SUSPENDED and LOST. When SysDisconnected is received, Curator sets
>> the state to SUSPENDED and then issues a sync() in the
>> background. This sync() does through Curator's normal connection
>> management, retries, etc. If the sync() doesn't succeed within the
>> connection timeout, Curator sets the state to LOST. How the
>> application handles these state changes is, of course, up to the
>> app.
> Sorry for the late reply.
>
> The connection timeout is defined in curator though? Or is taken from
> the session timeout? We're doing something like this now.
>
> -Ivan
Re: What should I do with SyncDisconnected
Posted by Ivan Kelly <iv...@apache.org>.
On Thu, Mar 14, 2013 at 02:43:58PM -0700, Jordan Zimmerman wrote:
> > But how long to wait? If the server is truly partitioned from ZK, then
> > I'll wait forever, and the client request will be hung forever.
> Curator waits for the time specified as the "connection
> timeout". Curator introduces three connection states, CONNECTED,
> SUSPENDED and LOST. When SysDisconnected is received, Curator sets
> the state to SUSPENDED and then issues a sync() in the
> background. This sync() does through Curator's normal connection
> management, retries, etc. If the sync() doesn't succeed within the
> connection timeout, Curator sets the state to LOST. How the
> application handles these state changes is, of course, up to the
> app.
Sorry for the late reply.
The connection timeout is defined in curator though? Or is taken from
the session timeout? We're doing something like this now.
-Ivan
Re: What should I do with SyncDisconnected
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
> But how long to wait? If the server is truly partitioned from ZK, then
> I'll wait forever, and the client request will be hung forever.
Curator waits for the time specified as the "connection timeout". Curator introduces three connection states, CONNECTED, SUSPENDED and LOST. When SysDisconnected is received, Curator sets the state to SUSPENDED and then issues a sync() in the background. This sync() does through Curator's normal connection management, retries, etc. If the sync() doesn't succeed within the connection timeout, Curator sets the state to LOST. How the application handles these state changes is, of course, up to the app.
-Jordan
Re: What should I do with SyncDisconnected
Posted by Ivan Kelly <iv...@apache.org>.
@kishore
> * First, How do clients know where to send the request, are they monitoring
> some ephemeral znodes in zookeeper. If yes, then after session timeout some
> other server should notice the ephemeral znode disappearing and recreate
> another ephemeral znode. Clients should not start sending request to new
> server, basically the server which is disconnected from zk permanently
> will never get any new requests.
The client looks up the owner of the partition before making a
request. I could have the client monitor ZK and cancel the request if
the owner changes. Alternatively, I could have a the client requests
timeout after a period and then recheck the ownership.
> * Second, how much tolerance do you have for downtime? if you are ok for
> not a server to serve a partition for X seconds ( session timeout), then on
> SyncDisconnected you can stop accepting requests and resume on
> SyncConnected.
The server will never receive SessionExpired though, so my server
needs to be aware of the session timeout. This is actually what we're
doing; Waiting X seconds for SyncConnected and if it doesn't come,
shutting down. This feels kludgy though.
@Jordan
> > SyncDisconnected can occur for a variety of reasons. It's in the class of
> > recoverable errors. Your app needs to go into a waiting state until
> > SysConnected is retrieved again or SessionExpired. Have you read
> > http://wiki.apache.org/hadoop/ZooKeeper/ErrorHandling ?
But how long to wait? If the server is truly partitioned from ZK, then
I'll wait forever, and the client request will be hung forever.
> >
> > You should consider using one of the high level ZooKeeper frameworks (such
> > as Curator which I wrote).
Conceptually the issue would still exist though. A high level library
doesn't solve the problem if the problem can't be solved with raw
zookeeper.
-Ivan
Re: What should I do with SyncDisconnected
Posted by kishore g <g....@gmail.com>.
Couple of questions,
* First, How do clients know where to send the request, are they monitoring
some ephemeral znodes in zookeeper. If yes, then after session timeout some
other server should notice the ephemeral znode disappearing and recreate
another ephemeral znode. Clients should not start sending request to new
server, basically the server which is disconnected from zk permanently
will never get any new requests.
* Second, how much tolerance do you have for downtime? if you are ok for
not a server to serve a partition for X seconds ( session timeout), then on
SyncDisconnected you can stop accepting requests and resume on
SyncConnected.
There are couple of other approaches to detect network partition, 1) try to
communicate with other nodes( some kind of gossip protocol) and see if they
can reach zk and based on the response put itself in limbo state etc or (2)
have a central co-ordinator and check with co-ordinator if it should
continue processing or not. None of these are trivial to implement
correctly.
On Wed, Mar 13, 2013 at 2:21 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:
> SyncDisconnected can occur for a variety of reasons. It's in the class of
> recoverable errors. Your app needs to go into a waiting state until
> SysConnected is retrieved again or SessionExpired. Have you read
> http://wiki.apache.org/hadoop/ZooKeeper/ErrorHandling ?
>
> You should consider using one of the high level ZooKeeper frameworks (such
> as Curator which I wrote).
>
> -Jordan
>
> On Mar 13, 2013, at 2:01 PM, Ivan Kelly <iv...@apache.org> wrote:
>
> > Hi guys,
> >
> > We have a usecase here where zookeeper is used to coordinate ownership
> > of partitions of a resource. When one server dies, the partition
> > should be moved to another server, etc. The action we need to take on
> > SessionExpired is very clear. We just kill the server.
> >
> > However it is unclear what we should do on a SyncDisconnected. We
> > can't just kill our server, as it may have just been one zookeeper
> > server failing. If we block all client requests to our server while we
> > wait for SyncConnected, we may block forever in the case that our
> > server is partitioned away from the zk cluster. If we continue to
> > serve requests, we risk split brain[1].
> >
> > What have people done in the past to resolve issues like this?
> >
> > -Ivan
> >
> >
> > [1] This is a risk anyhow without proper fencing, but a limited amount
> > is ok in our application.
>
>
Re: What should I do with SyncDisconnected
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
SyncDisconnected can occur for a variety of reasons. It's in the class of recoverable errors. Your app needs to go into a waiting state until SysConnected is retrieved again or SessionExpired. Have you read http://wiki.apache.org/hadoop/ZooKeeper/ErrorHandling ?
You should consider using one of the high level ZooKeeper frameworks (such as Curator which I wrote).
-Jordan
On Mar 13, 2013, at 2:01 PM, Ivan Kelly <iv...@apache.org> wrote:
> Hi guys,
>
> We have a usecase here where zookeeper is used to coordinate ownership
> of partitions of a resource. When one server dies, the partition
> should be moved to another server, etc. The action we need to take on
> SessionExpired is very clear. We just kill the server.
>
> However it is unclear what we should do on a SyncDisconnected. We
> can't just kill our server, as it may have just been one zookeeper
> server failing. If we block all client requests to our server while we
> wait for SyncConnected, we may block forever in the case that our
> server is partitioned away from the zk cluster. If we continue to
> serve requests, we risk split brain[1].
>
> What have people done in the past to resolve issues like this?
>
> -Ivan
>
>
> [1] This is a risk anyhow without proper fencing, but a limited amount
> is ok in our application.