You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Matthew Harrigan <ha...@gmail.com> on 2016/02/05 13:56:17 UTC
Using zookeeper to track started and complete tasks
Hello,
I posted a question
<http://stackoverflow.com/questions/35205591/using-zookeeper-to-manage-tasks-which-are-in-process-or-have-been-processed>
to stack overflow. Perhaps sending it to the zookeeper mailing list would
have been better. Any help/comments would be greatly appreciated.
After reading more about zookeeper, I suspect that one edge case which
could cause problems is if a worker loses connection after creating an
ephemeral node under started but before the client receives
acknowledgement. That worker would not know whether it was actually
created or not, and in the meantime some other worker could create the same
node. That would violate exactly once processing. Perhaps uuid's could
help. Maybe i could just restart the worker entirely if it loses
connection. Anyways that is a pretty nasty edge case and I really don't
need anything bulletproof.
Thank you
Re: Using zookeeper to track started and complete tasks
Posted by Patrick Hunt <ph...@apache.org>.
No worries. What you just said is correct, it honors that guarantee.
Keep in mind that even if the session that successfully created the
node (the 1 client) loses it's connection _during_ the creation the
znode may be created and no other session will then be able to create
it. That's what I was trying to point out re the faq reference. Hope
this helps.
Patrick
On Fri, Feb 5, 2016 at 4:07 PM, Matthew Harrigan
<ha...@gmail.com> wrote:
> The connection loss issue is really more of a tangent I inadvertently got
> started. My question relates to the guarantees zookeeper makes. Under any
> circumstance, can multiple clients create the same znode? Here I am
> talking about regular znodes, not ephemeral. Said more formally, if N
> clients simultaneously attempt to create the same znode (same path), will 0
> or 1 clients return successfully while the other clients either lose
> connection or throw a NodeExistsError? I suspect zookeeper does honor that
> guarantee but I don't see it mentioned here
> <http://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#ch_zkGuarantees>.
> My stackoverflow post was more complete than my email, I apologize for that.
>
> On Fri, Feb 5, 2016 at 6:15 PM, Patrick Hunt <ph...@apache.org> wrote:
>
>> See this item in the FAQ: "How should I handle the CONNECTION_LOSS
>> error?". If I understand you correctly it answers your question.
>> https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ
>>
>> Patrick
>>
>> On Fri, Feb 5, 2016 at 4:56 AM, Matthew Harrigan
>> <ha...@gmail.com> wrote:
>> > Hello,
>> >
>> > I posted a question
>> > <
>> http://stackoverflow.com/questions/35205591/using-zookeeper-to-manage-tasks-which-are-in-process-or-have-been-processed
>> >
>> > to stack overflow. Perhaps sending it to the zookeeper mailing list
>> would
>> > have been better. Any help/comments would be greatly appreciated.
>> >
>> > After reading more about zookeeper, I suspect that one edge case which
>> > could cause problems is if a worker loses connection after creating an
>> > ephemeral node under started but before the client receives
>> > acknowledgement. That worker would not know whether it was actually
>> > created or not, and in the meantime some other worker could create the
>> same
>> > node. That would violate exactly once processing. Perhaps uuid's could
>> > help. Maybe i could just restart the worker entirely if it loses
>> > connection. Anyways that is a pretty nasty edge case and I really don't
>> > need anything bulletproof.
>> >
>> > Thank you
>>
Re: Using zookeeper to track started and complete tasks
Posted by Matthew Harrigan <ha...@gmail.com>.
The connection loss issue is really more of a tangent I inadvertently got
started. My question relates to the guarantees zookeeper makes. Under any
circumstance, can multiple clients create the same znode? Here I am
talking about regular znodes, not ephemeral. Said more formally, if N
clients simultaneously attempt to create the same znode (same path), will 0
or 1 clients return successfully while the other clients either lose
connection or throw a NodeExistsError? I suspect zookeeper does honor that
guarantee but I don't see it mentioned here
<http://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#ch_zkGuarantees>.
My stackoverflow post was more complete than my email, I apologize for that.
On Fri, Feb 5, 2016 at 6:15 PM, Patrick Hunt <ph...@apache.org> wrote:
> See this item in the FAQ: "How should I handle the CONNECTION_LOSS
> error?". If I understand you correctly it answers your question.
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ
>
> Patrick
>
> On Fri, Feb 5, 2016 at 4:56 AM, Matthew Harrigan
> <ha...@gmail.com> wrote:
> > Hello,
> >
> > I posted a question
> > <
> http://stackoverflow.com/questions/35205591/using-zookeeper-to-manage-tasks-which-are-in-process-or-have-been-processed
> >
> > to stack overflow. Perhaps sending it to the zookeeper mailing list
> would
> > have been better. Any help/comments would be greatly appreciated.
> >
> > After reading more about zookeeper, I suspect that one edge case which
> > could cause problems is if a worker loses connection after creating an
> > ephemeral node under started but before the client receives
> > acknowledgement. That worker would not know whether it was actually
> > created or not, and in the meantime some other worker could create the
> same
> > node. That would violate exactly once processing. Perhaps uuid's could
> > help. Maybe i could just restart the worker entirely if it loses
> > connection. Anyways that is a pretty nasty edge case and I really don't
> > need anything bulletproof.
> >
> > Thank you
>
Re: Using zookeeper to track started and complete tasks
Posted by Patrick Hunt <ph...@apache.org>.
See this item in the FAQ: "How should I handle the CONNECTION_LOSS
error?". If I understand you correctly it answers your question.
https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ
Patrick
On Fri, Feb 5, 2016 at 4:56 AM, Matthew Harrigan
<ha...@gmail.com> wrote:
> Hello,
>
> I posted a question
> <http://stackoverflow.com/questions/35205591/using-zookeeper-to-manage-tasks-which-are-in-process-or-have-been-processed>
> to stack overflow. Perhaps sending it to the zookeeper mailing list would
> have been better. Any help/comments would be greatly appreciated.
>
> After reading more about zookeeper, I suspect that one edge case which
> could cause problems is if a worker loses connection after creating an
> ephemeral node under started but before the client receives
> acknowledgement. That worker would not know whether it was actually
> created or not, and in the meantime some other worker could create the same
> node. That would violate exactly once processing. Perhaps uuid's could
> help. Maybe i could just restart the worker entirely if it loses
> connection. Anyways that is a pretty nasty edge case and I really don't
> need anything bulletproof.
>
> Thank you