You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Kevin Burton <bu...@spinn3r.com> on 2009/01/07 01:39:35 UTC
Sending data during NodeDataChanged or NodeCreated
So if I understand this correctly, if I receive a NodeDataChanged event, and
then attempt do do a read of that node, there's a race condition where the
server could crash and I would be disconnected and my read would hit an
Exception
Or, the ACL could change and I no longer have permission to read the file
(though I did for a short window).
..... now I have to add all this logic to retry. Are there any other race
conditions I wonder.
Why not just send the byte[] data during the NodeDataChanged or NodeCreated
event from the server? This would avoid all these issues.
It's almost certainly what the user wants anyway.
Kevin
--
Founder/CEO Spinn3r.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com
Re: Sending data during NodeDataChanged or NodeCreated
Posted by Kevin Burton <bu...@spinn3r.com>.
> Perhaps a compromise would be augment the API, or your wrapper, such
> that it provides a Future<byte[]> which has more well-known semantics
> to clients.
>
Already there :)
I'll publish the source once my second pass is done and I believe I've
removed all the rope.
Kevin
--
Founder/CEO Spinn3r.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com
Re: Sending data during NodeDataChanged or NodeCreated
Posted by Adam Rosien <ad...@rosien.net>.
On Wed, Jan 7, 2009 at 12:46 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> On Wed, Jan 7, 2009 at 9:25 AM, Benjamin Reed <br...@yahoo-inc.com> wrote:
>
>> This is the behavior we had when we first implemented the API, and in every
>> case where people used the information there was a bug. it is virtually
>> impossible to use correctly. In general I'm all for giving people rope, but
>> if it always results in death, you should stop handing it out.
>>
>
> I think this is excessive rope...
>
> Most people want to read the data and having a race here is just asking for
> trouble.
>
> I'm not sure it is as much about excessive rope is it is about making it
> easy for users to stumble on the correct use case and reduce bugs.
>
> Ignorance is a wonderful gift you can give to your users :)
Perhaps a compromise would be augment the API, or your wrapper, such
that it provides a Future<byte[]> which has more well-known semantics
to clients.
.. Adam
Re: Sending data during NodeDataChanged or NodeCreated
Posted by Kevin Burton <bu...@spinn3r.com>.
On Thu, Jan 8, 2009 at 2:07 PM, Benjamin Reed <br...@yahoo-inc.com> wrote:
> if you do a getData("/a", true) and then "/a" changes, you will get a watch
> event. if "/a" changes again, you will not get an event. so, if you want to
> monitor "/a", you need to do a new getData() after each watch event to
> reregister the watch and get the new value. (re-registering watches on
> reconnect is a different issue. there are no disconnects in this example.)
>
> you are correct that zookeeper has some subtle things to watch out for.
> that is why we do not want to add more.
>
I honestly think they could all be removed if you implemented persistent
watches which don't require re-registration and add the data in the event.
I def don't want to add more though....
I'm not 100% convinced I'm right though as there may be other subtle ZK
things I"m missing but I'm like 90% sure :)
Kevin
--
Founder/CEO Spinn3r.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com
RE: Sending data during NodeDataChanged or NodeCreated
Posted by Benjamin Reed <br...@yahoo-inc.com>.
if you do a getData("/a", true) and then "/a" changes, you will get a watch event. if "/a" changes again, you will not get an event. so, if you want to monitor "/a", you need to do a new getData() after each watch event to reregister the watch and get the new value. (re-registering watches on reconnect is a different issue. there are no disconnects in this example.)
you are correct that zookeeper has some subtle things to watch out for. that is why we do not want to add more.
ben
-----Original Message-----
From: burtonator@gmail.com [mailto:burtonator@gmail.com] On Behalf Of Kevin Burton
Sent: Thursday, January 08, 2009 11:58 AM
To: zookeeper-user@hadoop.apache.org
Subject: Re: Sending data during NodeDataChanged or NodeCreated
>
>
> while the case in which a value only changes once, can be made slightly
> more optimal by passing the value in the watch event. it is not worth the
> risk. in our experience we had a application that was able to make that
> assumption initially and then later when the assumption became invalid it
> was very hard to diagnose.
>
I don't quite follow. In this scenario you would be sent two events, with
two pieces of data.
If ZK re-registers watches on reconnect, I don't see how it could be easier
than this.
> we don't want to make zookeeper harder to use by introducing mechanisms
> that only work with subtle assumptions.
>
I definitely think ZK has too much rope right now. It's far too easy to
make mistakes and there are lots of subtle undocumented behaviors.
Kevin
--
Founder/CEO Spinn3r.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com
Re: Sending data during NodeDataChanged or NodeCreated
Posted by Kevin Burton <bu...@spinn3r.com>.
>
>
> while the case in which a value only changes once, can be made slightly
> more optimal by passing the value in the watch event. it is not worth the
> risk. in our experience we had a application that was able to make that
> assumption initially and then later when the assumption became invalid it
> was very hard to diagnose.
>
I don't quite follow. In this scenario you would be sent two events, with
two pieces of data.
If ZK re-registers watches on reconnect, I don't see how it could be easier
than this.
> we don't want to make zookeeper harder to use by introducing mechanisms
> that only work with subtle assumptions.
>
I definitely think ZK has too much rope right now. It's far too easy to
make mistakes and there are lots of subtle undocumented behaviors.
Kevin
--
Founder/CEO Spinn3r.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com
RE: Sending data during NodeDataChanged or NodeCreated
Posted by Benjamin Reed <br...@yahoo-inc.com>.
I think you are looking at the race condition backwards. (and this btw is where the users had many of their bugs.) remember watches are one time triggers. so if you get a watch event it means a change has happened, so if /a is changed to 1 and then changed to 2. the data that will come back in the watch event will be 1, but the latest data will be 2.
if the application uses 1 from the watch event and runs with that, it will not be using the latest value. it will not even know that it isn't using the latest value since it will never get another watch event when /a is set to 2.
generally, applications use watches to monitor a value. in that case you never would want to use the value you get from the watch. instead, you use the value you get from the read that reregisters the watch.
while the case in which a value only changes once, can be made slightly more optimal by passing the value in the watch event. it is not worth the risk. in our experience we had a application that was able to make that assumption initially and then later when the assumption became invalid it was very hard to diagnose.
we don't want to make zookeeper harder to use by introducing mechanisms that only work with subtle assumptions.
ben
-----Original Message-----
From: burtonator@gmail.com [mailto:burtonator@gmail.com] On Behalf Of Kevin Burton
Sent: Wednesday, January 07, 2009 12:46 PM
To: zookeeper-user@hadoop.apache.org
Subject: Re: Sending data during NodeDataChanged or NodeCreated
On Wed, Jan 7, 2009 at 9:25 AM, Benjamin Reed <br...@yahoo-inc.com> wrote:
> This is the behavior we had when we first implemented the API, and in every
> case where people used the information there was a bug. it is virtually
> impossible to use correctly. In general I'm all for giving people rope, but
> if it always results in death, you should stop handing it out.
>
I think this is excessive rope...
Most people want to read the data and having a race here is just asking for
trouble.
I'm not sure it is as much about excessive rope is it is about making it
easy for users to stumble on the correct use case and reduce bugs.
Ignorance is a wonderful gift you can give to your users :)
>
> In your example, if the ACL changed and then the data changed, we would
> have a security hole if we sent the data with the watch.
>
I thought you might mention that. :) Technically there wouldn't be a
security hole if the operation was this:
- set foo to 'asdf'
- set ACL to foo blocking everyone reading it...
If you needed to prevent a read of 'asdf' you need to do this:
- set ACL to foo blocking everyone reading it...
- set foo to 'asdf'
When they are 1ms apart it's hard to understand but imagine if they were 10
hours apart.
Technically, there would be a 1ms window where clients could do a getData()
on the file and read the value.
Kevin
--
Founder/CEO Spinn3r.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com
Re: Sending data during NodeDataChanged or NodeCreated
Posted by Kevin Burton <bu...@spinn3r.com>.
On Wed, Jan 7, 2009 at 9:25 AM, Benjamin Reed <br...@yahoo-inc.com> wrote:
> This is the behavior we had when we first implemented the API, and in every
> case where people used the information there was a bug. it is virtually
> impossible to use correctly. In general I'm all for giving people rope, but
> if it always results in death, you should stop handing it out.
>
I think this is excessive rope...
Most people want to read the data and having a race here is just asking for
trouble.
I'm not sure it is as much about excessive rope is it is about making it
easy for users to stumble on the correct use case and reduce bugs.
Ignorance is a wonderful gift you can give to your users :)
>
> In your example, if the ACL changed and then the data changed, we would
> have a security hole if we sent the data with the watch.
>
I thought you might mention that. :) Technically there wouldn't be a
security hole if the operation was this:
- set foo to 'asdf'
- set ACL to foo blocking everyone reading it...
If you needed to prevent a read of 'asdf' you need to do this:
- set ACL to foo blocking everyone reading it...
- set foo to 'asdf'
When they are 1ms apart it's hard to understand but imagine if they were 10
hours apart.
Technically, there would be a 1ms window where clients could do a getData()
on the file and read the value.
Kevin
--
Founder/CEO Spinn3r.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com
RE: Sending data during NodeDataChanged or NodeCreated
Posted by Benjamin Reed <br...@yahoo-inc.com>.
This is the behavior we had when we first implemented the API, and in every case where people used the information there was a bug. it is virtually impossible to use correctly. In general I'm all for giving people rope, but if it always results in death, you should stop handing it out.
In your example, if the ACL changed and then the data changed, we would have a security hole if we sent the data with the watch.
ben
________________________________________
From: burtonator@gmail.com [burtonator@gmail.com] On Behalf Of Kevin Burton [burton@spinn3r.com]
Sent: Tuesday, January 06, 2009 4:39 PM
To: zookeeper-user@hadoop.apache.org
Subject: Sending data during NodeDataChanged or NodeCreated
So if I understand this correctly, if I receive a NodeDataChanged event, and
then attempt do do a read of that node, there's a race condition where the
server could crash and I would be disconnected and my read would hit an
Exception
Or, the ACL could change and I no longer have permission to read the file
(though I did for a short window).
..... now I have to add all this logic to retry. Are there any other race
conditions I wonder.
Why not just send the byte[] data during the NodeDataChanged or NodeCreated
event from the server? This would avoid all these issues.
It's almost certainly what the user wants anyway.
Kevin
--
Founder/CEO Spinn3r.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com
Re: Sending data during NodeDataChanged or NodeCreated
Posted by Kevin Burton <bu...@spinn3r.com>.
>
> > It's almost certainly what the user wants anyway.
> Its just that the watches are pretty lightweight and sending bytes around
> is
> just more work to do at the server. Though we should experiment with how
> much more load it generates and how useful would it be to send out the
> bytes
> with the event of nodedatachanged and nodecreated.
>
It might just be an alternate method... maybe not returning the data by
default.
It would solve the issue of deletes after NodeDataChange ....
Kevin
--
Founder/CEO Spinn3r.com
Location: San Francisco, CA
AIM/YIM: sfburtonator
Skype: burtonator
Work: http://spinn3r.com
Re: Sending data during NodeDataChanged or NodeCreated
Posted by Mahadev Konar <ma...@yahoo-inc.com>.
> So if I understand this correctly, if I receive a NodeDataChanged event, and
> then attempt do do a read of that node, there's a race condition where the
> server could crash and I would be disconnected and my read would hit an
> Exception
> Or, the ACL could change and I no longer have permission to read the file
> (though I did for a short window).
>
> ..... now I have to add all this logic to retry. Are there any other race
> conditions I wonder.
I think you have mentioned all of them.
>
> Why not just send the byte[] data during the NodeDataChanged or NodeCreated
> event from the server? This would avoid all these issues.
>
> It's almost certainly what the user wants anyway.
Its just that the watches are pretty lightweight and sending bytes around is
just more work to do at the server. Though we should experiment with how
much more load it generates and how useful would it be to send out the bytes
with the event of nodedatachanged and nodecreated.
mahadev
>
> Kevin