Retrying sequential znode creation

[Vishal K <vi...@gmail.com>]

Hi,

What is the best approach to have an idempotent create() operation for a
sequential node?

Suppose a client is trying to create a sequential node and it gets a
ConnectionLoss KeeperException, it cannot know for sure whether the request
succeeded or not. If in the meantime, the client's session is
re-established, the client would like to create a sequential znode again.
However, the client needs to know if its earlier request has succeeded or
not. If it did, then the client does not need to retry. To my understanding
ZooKeeper does not provide this feature. Can someone confirm this?

External to ZooKeeper, the client can either set a unique UUID in the path
to the create call or write the UUID as part of its data. Before retrying,
it can read back all the children of the parent znode and go through the
list to determine if its earlier request had succeeded. This doesn't sound
that appealing to me.

I am guessing this is a common problem that many would have faced. Can folks
give a feedback on what their approach was?

Thanks.
-Vishal

Re: Retrying sequential znode creation

[Patrick Hunt <ph...@apache.org>]

Hi Ted, Mahadev is in the best position to comment (he looked at it last)
but iirc when we started looking into implementing this we immediately ran
into so big questions. One was what to do if the logs had been cleaned up
and the individual transactions no longer available. This could be overcome
by changes wrt cleanup, log rotation, etc... There was another more
bulletproof option, essentially to keep all the changes in memory that might
be necessary to implement 22, however this might mean a significant increase
in mem requirements and general bookkeeping. It turned out (again correct me
if I'm wrong) that more thought was going to be necessary, esp around
ensuring correct operation in any/all special cases.

Patrick

On Wed, Oct 13, 2010 at 12:49 PM, Ted Dunning <te...@gmail.com> wrote:

> Patrick,
>
> What are these hurdles?  The last comment on ZK-22 was last winter.  Back
> then, it didn't sound like
> it was going to be that hard.
>
> On Wed, Oct 13, 2010 at 12:08 PM, Patrick Hunt <ph...@apache.org> wrote:
>
> > 22 would help with this issue
> > https://issues.apache.org/jira/browse/ZOOKEEPER-22
> > however there are some real hurdles to implementing 22 successfully.
> >
>

Re: Retrying sequential znode creation

[Patrick Hunt <ph...@apache.org>]

On Wed, Oct 20, 2010 at 3:27 PM, Ted Dunning <te...@gmail.com> wrote:

> These corner cases are relatively rare, I would think (I personally keep
> logs around for days or longer).
>

A concern I would have is that it does add complexity, would be hard to
debug...


> Would it be possible to get a partial solution in place that invokes the
> current behavior if logs aren't available?


Seems like it's possible. The issue of finding a viable solution (one where,
for example, the memory overhead is limited) is still an issue though. In
the end it wouldn't really help the end user, given they would still have to
code for this corner case.

Patrick


> On Wed, Oct 20, 2010 at 10:42 AM, Patrick Hunt <ph...@gmail.com> wrote:
>
> > Hi Ted, Mahadev is in the best position to comment (he looked at it last)
> > but iirc when we started looking into implementing this we immediately
> ran
> > into so big questions. One was what to do if the logs had been cleaned up
> > and the individual transactions no longer available. This could be
> overcome
> > by changes wrt cleanup, log rotation, etc... There was another more
> > bulletproof option, essentially to keep all the changes in memory that
> > might
> > be necessary to implement 22, however this might mean a significant
> > increase
> > in mem requirements and general bookkeeping. It turned out (again correct
> > me
> > if I'm wrong) that more thought was going to be necessary, esp around
> > ensuring correct operation in any/all special cases.
> >
> > Patrick
> >
> > On Wed, Oct 13, 2010 at 12:49 PM, Ted Dunning <te...@gmail.com>
> > wrote:
> >
> > > Patrick,
> > >
> > > What are these hurdles?  The last comment on ZK-22 was last winter.
>  Back
> > > then, it didn't sound like
> > > it was going to be that hard.
> > >
> > > On Wed, Oct 13, 2010 at 12:08 PM, Patrick Hunt <ph...@apache.org>
> wrote:
> > >
> > > > 22 would help with this issue
> > > > https://issues.apache.org/jira/browse/ZOOKEEPER-22
> > > > however there are some real hurdles to implementing 22 successfully.
> > > >
> > >
> >
>

Re: Retrying sequential znode creation

[Ted Dunning <te...@gmail.com>]

These corner cases are relatively rare, I would think (I personally keep
logs around for days or longer).

Would it be possible to get a partial solution in place that invokes the
current behavior if logs aren't available?

On Wed, Oct 20, 2010 at 10:42 AM, Patrick Hunt <ph...@gmail.com> wrote:

> Hi Ted, Mahadev is in the best position to comment (he looked at it last)
> but iirc when we started looking into implementing this we immediately ran
> into so big questions. One was what to do if the logs had been cleaned up
> and the individual transactions no longer available. This could be overcome
> by changes wrt cleanup, log rotation, etc... There was another more
> bulletproof option, essentially to keep all the changes in memory that
> might
> be necessary to implement 22, however this might mean a significant
> increase
> in mem requirements and general bookkeeping. It turned out (again correct
> me
> if I'm wrong) that more thought was going to be necessary, esp around
> ensuring correct operation in any/all special cases.
>
> Patrick
>
> On Wed, Oct 13, 2010 at 12:49 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > Patrick,
> >
> > What are these hurdles?  The last comment on ZK-22 was last winter.  Back
> > then, it didn't sound like
> > it was going to be that hard.
> >
> > On Wed, Oct 13, 2010 at 12:08 PM, Patrick Hunt <ph...@apache.org> wrote:
> >
> > > 22 would help with this issue
> > > https://issues.apache.org/jira/browse/ZOOKEEPER-22
> > > however there are some real hurdles to implementing 22 successfully.
> > >
> >
>

Re: Retrying sequential znode creation

[Patrick Hunt <ph...@gmail.com>]

Hi Ted, Mahadev is in the best position to comment (he looked at it last)
but iirc when we started looking into implementing this we immediately ran
into so big questions. One was what to do if the logs had been cleaned up
and the individual transactions no longer available. This could be overcome
by changes wrt cleanup, log rotation, etc... There was another more
bulletproof option, essentially to keep all the changes in memory that might
be necessary to implement 22, however this might mean a significant increase
in mem requirements and general bookkeeping. It turned out (again correct me
if I'm wrong) that more thought was going to be necessary, esp around
ensuring correct operation in any/all special cases.

Patrick

On Wed, Oct 13, 2010 at 12:49 PM, Ted Dunning <te...@gmail.com> wrote:

> Patrick,
>
> What are these hurdles?  The last comment on ZK-22 was last winter.  Back
> then, it didn't sound like
> it was going to be that hard.
>
> On Wed, Oct 13, 2010 at 12:08 PM, Patrick Hunt <ph...@apache.org> wrote:
>
> > 22 would help with this issue
> > https://issues.apache.org/jira/browse/ZOOKEEPER-22
> > however there are some real hurdles to implementing 22 successfully.
> >
>

Re: Retrying sequential znode creation

[Ted Dunning <te...@gmail.com>]

Patrick,

What are these hurdles?  The last comment on ZK-22 was last winter.  Back
then, it didn't sound like
it was going to be that hard.

On Wed, Oct 13, 2010 at 12:08 PM, Patrick Hunt <ph...@apache.org> wrote:

> 22 would help with this issue
> https://issues.apache.org/jira/browse/ZOOKEEPER-22
> however there are some real hurdles to implementing 22 successfully.
>

Re: Retrying sequential znode creation

[Patrick Hunt <ph...@apache.org>]

On Wed, Oct 13, 2010 at 5:58 AM, Vishal K <vi...@gmail.com> wrote:
>
> However, gets trickier because there is no explicit way (to my knowledge)
> to
> get CreateMode for a znode. As a result, we cannot tell whether a node is
> sequential or not.
>

Sequentials are really just regular znodes with fancy naming applied by the
cluster at create time, subsequently it makes no distinction. Using the
format of the name would be the be only/best way I know if you want to
distinguish yourself. (or put some data into the znode itself)

22 would help with this issue
https://issues.apache.org/jira/browse/ZOOKEEPER-22
however there are some real hurdles to implementing 22 successfully.

Patrick


>
> Thanks.
> -Vishal
>
>
> On Tue, Oct 12, 2010 at 5:36 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > Yes.  This is indeed a problem.  I generally try to avoid sequential
> nodes
> > unless they are ephemeral and if I get an error on
> > creation, I generally have to either tear down the connection (losing all
> > other ephemeral nodes in the process) or scan through
> > all live nodes trying to determine if mine got created.  Neither is a
> very
> > acceptable answer so I try to avoid the problem.
> >
> > Your UUID answer is one option.  At least you know what file got created
> > (or
> > not) and with good naming you can pretty much guarantee no collisions.
>  You
> > don't have to scan all children since you can simply check for the
> > existence
> > of the file of interest.
> >
> > There was a JIRA filed that was supposed to take care of this problem,
> but
> > I
> > don't know the state of play there.
> >
> > On Tue, Oct 12, 2010 at 12:11 PM, Vishal K <vi...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > What is the best approach to have an idempotent create() operation for
> a
> > > sequential node?
> > >
> > > Suppose a client is trying to create a sequential node and it gets a
> > > ConnectionLoss KeeperException, it cannot know for sure whether the
> > request
> > > succeeded or not. If in the meantime, the client's session is
> > > re-established, the client would like to create a sequential znode
> again.
> > > However, the client needs to know if its earlier request has succeeded
> or
> > > not. If it did, then the client does not need to retry. To my
> > understanding
> > > ZooKeeper does not provide this feature. Can someone confirm this?
> > >
> > > External to ZooKeeper, the client can either set a unique UUID in the
> > path
> > > to the create call or write the UUID as part of its data. Before
> > retrying,
> > > it can read back all the children of the parent znode and go through
> the
> > > list to determine if its earlier request had succeeded. This doesn't
> > sound
> > > that appealing to me.
> > >
> > > I am guessing this is a common problem that many would have faced. Can
> > > folks
> > > give a feedback on what their approach was?
> > >
> > > Thanks.
> > > -Vishal
> > >
> >
>

Re: Retrying sequential znode creation

[Vishal K <vi...@gmail.com>]

Hi Ted,

Thanks for the reply. I prefer to store the UUID in the data itself. I
thought about storing it in znode names, but the downside of this approach
is that when a Watcher receives an event, it needs to strip off the UUID.
Otherwise, it wont be possible to figure out the order znodes (e.g., using
sort()). Also, in our case, there are very few sequential children. So
listing and reading all children is not a big overhead.

However, gets trickier because there is no explicit way (to my knowledge) to
get CreateMode for a znode. As a result, we cannot tell whether a node is
sequential or not.

Thanks.
-Vishal


On Tue, Oct 12, 2010 at 5:36 PM, Ted Dunning <te...@gmail.com> wrote:

> Yes.  This is indeed a problem.  I generally try to avoid sequential nodes
> unless they are ephemeral and if I get an error on
> creation, I generally have to either tear down the connection (losing all
> other ephemeral nodes in the process) or scan through
> all live nodes trying to determine if mine got created.  Neither is a very
> acceptable answer so I try to avoid the problem.
>
> Your UUID answer is one option.  At least you know what file got created
> (or
> not) and with good naming you can pretty much guarantee no collisions.  You
> don't have to scan all children since you can simply check for the
> existence
> of the file of interest.
>
> There was a JIRA filed that was supposed to take care of this problem, but
> I
> don't know the state of play there.
>
> On Tue, Oct 12, 2010 at 12:11 PM, Vishal K <vi...@gmail.com> wrote:
>
> > Hi,
> >
> > What is the best approach to have an idempotent create() operation for a
> > sequential node?
> >
> > Suppose a client is trying to create a sequential node and it gets a
> > ConnectionLoss KeeperException, it cannot know for sure whether the
> request
> > succeeded or not. If in the meantime, the client's session is
> > re-established, the client would like to create a sequential znode again.
> > However, the client needs to know if its earlier request has succeeded or
> > not. If it did, then the client does not need to retry. To my
> understanding
> > ZooKeeper does not provide this feature. Can someone confirm this?
> >
> > External to ZooKeeper, the client can either set a unique UUID in the
> path
> > to the create call or write the UUID as part of its data. Before
> retrying,
> > it can read back all the children of the parent znode and go through the
> > list to determine if its earlier request had succeeded. This doesn't
> sound
> > that appealing to me.
> >
> > I am guessing this is a common problem that many would have faced. Can
> > folks
> > give a feedback on what their approach was?
> >
> > Thanks.
> > -Vishal
> >
>

Re: Retrying sequential znode creation

[Ted Dunning <te...@gmail.com>]

Yes.  This is indeed a problem.  I generally try to avoid sequential nodes
unless they are ephemeral and if I get an error on
creation, I generally have to either tear down the connection (losing all
other ephemeral nodes in the process) or scan through
all live nodes trying to determine if mine got created.  Neither is a very
acceptable answer so I try to avoid the problem.

Your UUID answer is one option.  At least you know what file got created (or
not) and with good naming you can pretty much guarantee no collisions.  You
don't have to scan all children since you can simply check for the existence
of the file of interest.

There was a JIRA filed that was supposed to take care of this problem, but I
don't know the state of play there.

On Tue, Oct 12, 2010 at 12:11 PM, Vishal K <vi...@gmail.com> wrote:

> Hi,
>
> What is the best approach to have an idempotent create() operation for a
> sequential node?
>
> Suppose a client is trying to create a sequential node and it gets a
> ConnectionLoss KeeperException, it cannot know for sure whether the request
> succeeded or not. If in the meantime, the client's session is
> re-established, the client would like to create a sequential znode again.
> However, the client needs to know if its earlier request has succeeded or
> not. If it did, then the client does not need to retry. To my understanding
> ZooKeeper does not provide this feature. Can someone confirm this?
>
> External to ZooKeeper, the client can either set a unique UUID in the path
> to the create call or write the UUID as part of its data. Before retrying,
> it can read back all the children of the parent znode and go through the
> list to determine if its earlier request had succeeded. This doesn't sound
> that appealing to me.
>
> I am guessing this is a common problem that many would have faced. Can
> folks
> give a feedback on what their approach was?
>
> Thanks.
> -Vishal
>