You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Joe Gamache <ga...@cabotresearch.com> on 2012/05/16 17:59:14 UTC
Multi question on using previous ops results
I need to do something like the following:
Stat stats = nodeToSet.getStat();
if (stats != null) {
int version = stats.getVersion();
stats = client.getZk().setData(path, data, version);
(data was preset in code not shown). Since I have multiple processes
all vying to do things, in the worst case (which happens too
frequently), two processes execute the above at basically the same time,
which causes errors. I thought I might be able to use Multi to do this
atomically, but I cannot figure out how to use the result (the version
number above) from one Op in a later Op. Is this possible?
This was the simplest example I could find, but this happens often -
usually with Paths. For instance, sometimes I create a zookeeper node
and want a "LOCK" node underneath it. If I follow the "recipe" and use
an ephemeral sequential node as the lock, how can I refer to sequence in
a Path in subsequent Op added to a "multi" or is there any way to do it
atomically? Sorry that was so very wordy. What I was really trying to
ask, is how do I create and lock a node atomically?
thanks for any help, insight!
joe
Re: Multi question on using previous ops results
Posted by Joe Gamache <ga...@cabotresearch.com>.
As far as I can tell, the "under construction" approach will let me do
everything I need! Thanks - that is a simple and elegant approach that
I probably should have known about, at least I know now....
On 5/18/2012 8:03 PM, Ted Dunning wrote:
> On Fri, May 18, 2012 at 1:55 PM, Joe Gamache<ga...@cabotresearch.com>wrote:
>
>> ...
>>
>> I have a one process that is trying to create some "work" that will be
>> represented via a ZNode. A large number of other processes are looking
>> for work. The node that represents the work will be sequential. Most
>> importantly, it will be incomplete until the structure below it is
>> completed.
>
> That is easy. Just create the lower structures first. If the upper
> structure needs to have an indicator that it is under construction, just
> set a bit inside it using the atomic update method. Or create a separate
> node to do leader election.
>
>
>> Thus, I would like to be able to create that node and lock it atomically
>> OR create the entire structure below it atomically.
>
> Remember ZK doesn't do locks.
>
> You can create the node with the "under-construction" bit set. Or you can
> use a multi to create the parent and children simultaneously. You can
> create the parent with references to a version in another node by getting
> the version of the parent, getting the versions of the children, updating
> the parent with the version specified. Retry with backoff on failure.
>
> I would like to avoid hacking in another parallel node structure solely for
>> the purpose of locking (as there will be many 1000's of these created)
>>
> As you like. This can be handled pretty easily. Setting watches on the
> release of a node is a bit trickier, but can be handled as well.
>
>
>
>> All I am really wondering is can Zookeeper do this? If not, I can find
>> alternate ways around - they just seem less elegant to me.
>>
> It can.
>
Re: Multi question on using previous ops results
Posted by Ted Dunning <te...@gmail.com>.
On Fri, May 18, 2012 at 1:55 PM, Joe Gamache <ga...@cabotresearch.com>wrote:
> ...
>
> I have a one process that is trying to create some "work" that will be
> represented via a ZNode. A large number of other processes are looking
> for work. The node that represents the work will be sequential. Most
> importantly, it will be incomplete until the structure below it is
> completed.
That is easy. Just create the lower structures first. If the upper
structure needs to have an indicator that it is under construction, just
set a bit inside it using the atomic update method. Or create a separate
node to do leader election.
> Thus, I would like to be able to create that node and lock it atomically
> OR create the entire structure below it atomically.
Remember ZK doesn't do locks.
You can create the node with the "under-construction" bit set. Or you can
use a multi to create the parent and children simultaneously. You can
create the parent with references to a version in another node by getting
the version of the parent, getting the versions of the children, updating
the parent with the version specified. Retry with backoff on failure.
I would like to avoid hacking in another parallel node structure solely for
> the purpose of locking (as there will be many 1000's of these created)
>
As you like. This can be handled pretty easily. Setting watches on the
release of a node is a bit trickier, but can be handled as well.
>
> All I am really wondering is can Zookeeper do this? If not, I can find
> alternate ways around - they just seem less elegant to me.
>
It can.
Re: Multi question on using previous ops results
Posted by Joe Gamache <ga...@cabotresearch.com>.
Can you say again on a higher level what you really want to do?
Hmmm. Probably not, but I will still try :)
I have a one process that is trying to create some "work" that will be
represented via a ZNode. A large number of other processes are looking
for work. The node that represents the work will be sequential. Most
importantly, it will be incomplete until the structure below it is
completed. Thus, I would like to be able to create that node and lock
it atomically OR create the entire structure below it atomically. I
would like to avoid hacking in another parallel node structure solely
for the purpose of locking (as there will be many 1000's of these created)
All I am really wondering is can Zookeeper do this? If not, I can find
alternate ways around - they just seem less elegant to me.
On 5/18/2012 12:46 PM, Ted Dunning wrote:
> No. It still isn't clear what benefit is of putting the version number of
> one field into another.
>
> You seem to be focussing in on the minutiae of your proposed method rather
> than on the goal that you really want to solve.
>
> A problem that seems similar to what you want is to form a graph where A
> refers to B refers to C refers to A. The desired option would be update
> the graph atomically so that B now points to D which points to A. The
> intended use of multi here would be to do the standard read-modify-write
> sort of cycle where you read the contents and versions of the items to be
> updated or referenced in one operation and then in the next you do an
> update which is conditioned on version with items which must remain static
> having a version check. You repeat this cycle as many times as it takes,
> presumably backing off between attempts.
>
> This has a theoretical possibility of fast updaters starving slow updaters
> but this has never even been a possibility in any system I have seen,
> largely because update rates are typically pretty low.
>
> Can you say again on a higher level what you really want to do?
>
> On Fri, May 18, 2012 at 11:43 AM, Joe Gamache<ga...@cabotresearch.com>wrote:
>
>> ...
>> Ideally, I would like to be able to replace the "0" in various "setData"
>> calls, not sure how to do that exactly, but the above would be a decent
>> start. [If a node already existed and I was calling the Op.setData method
>> and wanted to follow that with a Op.check call, I believe that I currently
>> have no way of knowing the version number....]
>>
>> Does this make more sense than the gibberish I was trying below?
>>
>> thanks,
>>
>> Joe
>>
>>
>>
>> On 5/16/2012 8:06 PM, Joe Gamache wrote:
>>
>>> No thought about two znodes being synchronized at all.
>>>
>>> The higher level intent of the first (code) example is: how do you get
>>> the version number for a set operation and use that value in the set
>>> atomically? For the second one it is: how to I create and lock a znode
>>> atomically? [maybe the Lock znode is attached to the node I am creating,
>>> maybe in another structure? I don't really care, just can I do it
>>> atomically?]
>>>
>>> Another way to do what I would like would be to create a znode structure
>>> that is not "attached" to the existing one at all, then once I have it the
>>> way it should be, attach it to an existing node. However, I am quite
>>> certain this is not possible....
>>>
>>>
>>>
>>> On 5/16/2012 7:21 PM, Ted Dunning wrote:
>>>
>>>> It looks like you are assuming that the versions of two znodes are
>>>> synchronized. That seems pretty dangerous.
>>>>
>>>> What is the higher level intent here? Would it better to simply build a
>>>> multi to update both znodes?
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On May 16, 2012, at 11:59 AM, Joe Gamache<ga...@cabotresearch.com>>
>>>> wrote:
>>>>
>>>> I need to do something like the following:
>>>>> Stat stats = nodeToSet.getStat();
>>>>> if (stats != null) {
>>>>> int version = stats.getVersion();
>>>>> stats = client.getZk().setData(path, data, version);
>>>>>
>>>>> (data was preset in code not shown). Since I have multiple processes
>>>>> all vying to do things, in the worst case (which happens too frequently),
>>>>> two processes execute the above at basically the same time, which causes
>>>>> errors. I thought I might be able to use Multi to do this atomically, but
>>>>> I cannot figure out how to use the result (the version number above) from
>>>>> one Op in a later Op. Is this possible?
>>>>>
>>>>> This was the simplest example I could find, but this happens often -
>>>>> usually with Paths. For instance, sometimes I create a zookeeper node and
>>>>> want a "LOCK" node underneath it. If I follow the "recipe" and use an
>>>>> ephemeral sequential node as the lock, how can I refer to sequence in a
>>>>> Path in subsequent Op added to a "multi" or is there any way to do it
>>>>> atomically? Sorry that was so very wordy. What I was really trying to
>>>>> ask, is how do I create and lock a node atomically?
>>>>>
>>>>> thanks for any help, insight!
>>>>>
>>>>> joe
>>>>>
>>>>>
Re: Multi question on using previous ops results
Posted by Ted Dunning <te...@gmail.com>.
No. It still isn't clear what benefit is of putting the version number of
one field into another.
You seem to be focussing in on the minutiae of your proposed method rather
than on the goal that you really want to solve.
A problem that seems similar to what you want is to form a graph where A
refers to B refers to C refers to A. The desired option would be update
the graph atomically so that B now points to D which points to A. The
intended use of multi here would be to do the standard read-modify-write
sort of cycle where you read the contents and versions of the items to be
updated or referenced in one operation and then in the next you do an
update which is conditioned on version with items which must remain static
having a version check. You repeat this cycle as many times as it takes,
presumably backing off between attempts.
This has a theoretical possibility of fast updaters starving slow updaters
but this has never even been a possibility in any system I have seen,
largely because update rates are typically pretty low.
Can you say again on a higher level what you really want to do?
On Fri, May 18, 2012 at 11:43 AM, Joe Gamache <ga...@cabotresearch.com>wrote:
> ...
> Ideally, I would like to be able to replace the "0" in various "setData"
> calls, not sure how to do that exactly, but the above would be a decent
> start. [If a node already existed and I was calling the Op.setData method
> and wanted to follow that with a Op.check call, I believe that I currently
> have no way of knowing the version number....]
>
> Does this make more sense than the gibberish I was trying below?
>
> thanks,
>
> Joe
>
>
>
> On 5/16/2012 8:06 PM, Joe Gamache wrote:
>
>> No thought about two znodes being synchronized at all.
>>
>> The higher level intent of the first (code) example is: how do you get
>> the version number for a set operation and use that value in the set
>> atomically? For the second one it is: how to I create and lock a znode
>> atomically? [maybe the Lock znode is attached to the node I am creating,
>> maybe in another structure? I don't really care, just can I do it
>> atomically?]
>>
>> Another way to do what I would like would be to create a znode structure
>> that is not "attached" to the existing one at all, then once I have it the
>> way it should be, attach it to an existing node. However, I am quite
>> certain this is not possible....
>>
>>
>>
>> On 5/16/2012 7:21 PM, Ted Dunning wrote:
>>
>>> It looks like you are assuming that the versions of two znodes are
>>> synchronized. That seems pretty dangerous.
>>>
>>> What is the higher level intent here? Would it better to simply build a
>>> multi to update both znodes?
>>>
>>> Sent from my iPhone
>>>
>>> On May 16, 2012, at 11:59 AM, Joe Gamache<ga...@cabotresearch.com>>
>>> wrote:
>>>
>>> I need to do something like the following:
>>>>
>>>> Stat stats = nodeToSet.getStat();
>>>> if (stats != null) {
>>>> int version = stats.getVersion();
>>>> stats = client.getZk().setData(path, data, version);
>>>>
>>>> (data was preset in code not shown). Since I have multiple processes
>>>> all vying to do things, in the worst case (which happens too frequently),
>>>> two processes execute the above at basically the same time, which causes
>>>> errors. I thought I might be able to use Multi to do this atomically, but
>>>> I cannot figure out how to use the result (the version number above) from
>>>> one Op in a later Op. Is this possible?
>>>>
>>>> This was the simplest example I could find, but this happens often -
>>>> usually with Paths. For instance, sometimes I create a zookeeper node and
>>>> want a "LOCK" node underneath it. If I follow the "recipe" and use an
>>>> ephemeral sequential node as the lock, how can I refer to sequence in a
>>>> Path in subsequent Op added to a "multi" or is there any way to do it
>>>> atomically? Sorry that was so very wordy. What I was really trying to
>>>> ask, is how do I create and lock a node atomically?
>>>>
>>>> thanks for any help, insight!
>>>>
>>>> joe
>>>>
>>>>
>>
>
Re: Multi question on using previous ops results
Posted by Joe Gamache <ga...@cabotresearch.com>.
Hi,
I have looked at this in more detail. I found this thread:
http://zookeeper-user.578899.n2.nabble.com/Create-nested-paths-td7331156.html
which said: "Sample usage can be found in
org.apache.zookeeper.test.MultiTransactionTest"
Using part of that code to illustrate. What I would like to be able to
do is something akin to:
Integer REF1;
Integer REF2;
Integer REF3;
zk.multi(Arrays.asList(
/* Create */
REF1 = Op.create("/multi", null, new byte[0],
Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT_SEQUENTIAL),
Op.setData("", REF1, "DATA_VALUE".getBytes(), 0),
REF2 = Op.create("/a", REF1, new byte[0],
Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT),
REF3 = Op.create("/1", REF2 new byte[0],
Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT),
Obviously, use of Integer is just for illustration and the CAPs are for
effect. However, with this type of API the "multi" implementation
could, for a single input list, maintain a structure of previous Op's
that could then be referred to in future ops. When a "reference" was
passed in that was non-null, it would just that reference in some
intelligent way. Above, the use of REF1 on the "setData" method would
lookup the path created for REF1 and use that to set the data. The use
of REF1 in the Op.create would APPEND the input path to the path created
for REF1. Thus, the REF2 line might create a path like /multi_00000001/a
Ideally, I would like to be able to replace the "0" in various "setData"
calls, not sure how to do that exactly, but the above would be a decent
start. [If a node already existed and I was calling the Op.setData
method and wanted to follow that with a Op.check call, I believe that I
currently have no way of knowing the version number....]
Does this make more sense than the gibberish I was trying below?
thanks,
Joe
On 5/16/2012 8:06 PM, Joe Gamache wrote:
> No thought about two znodes being synchronized at all.
>
> The higher level intent of the first (code) example is: how do you get
> the version number for a set operation and use that value in the set
> atomically? For the second one it is: how to I create and lock a
> znode atomically? [maybe the Lock znode is attached to the node I am
> creating, maybe in another structure? I don't really care, just can I
> do it atomically?]
>
> Another way to do what I would like would be to create a znode
> structure that is not "attached" to the existing one at all, then once
> I have it the way it should be, attach it to an existing node.
> However, I am quite certain this is not possible....
>
>
>
> On 5/16/2012 7:21 PM, Ted Dunning wrote:
>> It looks like you are assuming that the versions of two znodes are
>> synchronized. That seems pretty dangerous.
>>
>> What is the higher level intent here? Would it better to simply
>> build a multi to update both znodes?
>>
>> Sent from my iPhone
>>
>> On May 16, 2012, at 11:59 AM, Joe Gamache<ga...@cabotresearch.com>
>> wrote:
>>
>>> I need to do something like the following:
>>>
>>> Stat stats = nodeToSet.getStat();
>>> if (stats != null) {
>>> int version = stats.getVersion();
>>> stats = client.getZk().setData(path, data, version);
>>>
>>> (data was preset in code not shown). Since I have multiple
>>> processes all vying to do things, in the worst case (which happens
>>> too frequently), two processes execute the above at basically the
>>> same time, which causes errors. I thought I might be able to use
>>> Multi to do this atomically, but I cannot figure out how to use the
>>> result (the version number above) from one Op in a later Op. Is
>>> this possible?
>>>
>>> This was the simplest example I could find, but this happens often -
>>> usually with Paths. For instance, sometimes I create a zookeeper
>>> node and want a "LOCK" node underneath it. If I follow the "recipe"
>>> and use an ephemeral sequential node as the lock, how can I refer to
>>> sequence in a Path in subsequent Op added to a "multi" or is there
>>> any way to do it atomically? Sorry that was so very wordy. What I
>>> was really trying to ask, is how do I create and lock a node
>>> atomically?
>>>
>>> thanks for any help, insight!
>>>
>>> joe
>>>
>
Re: Multi question on using previous ops results
Posted by Joe Gamache <ga...@cabotresearch.com>.
No thought about two znodes being synchronized at all.
The higher level intent of the first (code) example is: how do you get
the version number for a set operation and use that value in the set
atomically? For the second one it is: how to I create and lock a znode
atomically? [maybe the Lock znode is attached to the node I am
creating, maybe in another structure? I don't really care, just can I
do it atomically?]
Another way to do what I would like would be to create a znode structure
that is not "attached" to the existing one at all, then once I have it
the way it should be, attach it to an existing node. However, I am
quite certain this is not possible....
On 5/16/2012 7:21 PM, Ted Dunning wrote:
> It looks like you are assuming that the versions of two znodes are synchronized. That seems pretty dangerous.
>
> What is the higher level intent here? Would it better to simply build a multi to update both znodes?
>
> Sent from my iPhone
>
> On May 16, 2012, at 11:59 AM, Joe Gamache<ga...@cabotresearch.com> wrote:
>
>> I need to do something like the following:
>>
>> Stat stats = nodeToSet.getStat();
>> if (stats != null) {
>> int version = stats.getVersion();
>> stats = client.getZk().setData(path, data, version);
>>
>> (data was preset in code not shown). Since I have multiple processes all vying to do things, in the worst case (which happens too frequently), two processes execute the above at basically the same time, which causes errors. I thought I might be able to use Multi to do this atomically, but I cannot figure out how to use the result (the version number above) from one Op in a later Op. Is this possible?
>>
>> This was the simplest example I could find, but this happens often - usually with Paths. For instance, sometimes I create a zookeeper node and want a "LOCK" node underneath it. If I follow the "recipe" and use an ephemeral sequential node as the lock, how can I refer to sequence in a Path in subsequent Op added to a "multi" or is there any way to do it atomically? Sorry that was so very wordy. What I was really trying to ask, is how do I create and lock a node atomically?
>>
>> thanks for any help, insight!
>>
>> joe
>>
Re: Multi question on using previous ops results
Posted by Ted Dunning <te...@gmail.com>.
It looks like you are assuming that the versions of two znodes are synchronized. That seems pretty dangerous.
What is the higher level intent here? Would it better to simply build a multi to update both znodes?
Sent from my iPhone
On May 16, 2012, at 11:59 AM, Joe Gamache <ga...@cabotresearch.com> wrote:
> I need to do something like the following:
>
> Stat stats = nodeToSet.getStat();
> if (stats != null) {
> int version = stats.getVersion();
> stats = client.getZk().setData(path, data, version);
>
> (data was preset in code not shown). Since I have multiple processes all vying to do things, in the worst case (which happens too frequently), two processes execute the above at basically the same time, which causes errors. I thought I might be able to use Multi to do this atomically, but I cannot figure out how to use the result (the version number above) from one Op in a later Op. Is this possible?
>
> This was the simplest example I could find, but this happens often - usually with Paths. For instance, sometimes I create a zookeeper node and want a "LOCK" node underneath it. If I follow the "recipe" and use an ephemeral sequential node as the lock, how can I refer to sequence in a Path in subsequent Op added to a "multi" or is there any way to do it atomically? Sorry that was so very wordy. What I was really trying to ask, is how do I create and lock a node atomically?
>
> thanks for any help, insight!
>
> joe
>