You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Thomas Koch <th...@koch.ro> on 2011/11/01 17:40:06 UTC
Immutable DataTree
Hi,
I'm like two days away from having a version of ZooKeeper with an immutable
DataTree on the server. This means that for any modification of the DataTree,
all Nodes from the modified one up to the root are replaced by new nodes.
Advantages:
- No synchronisation needed when accessing the DataTree.
- The snapshotter thread gets an immutable datatree and will write a
consistent DataTree to the disk.
- No headaches whether multi transactions could lead to issues with
(de)serialization.
- Much better testability.
- No concurrency - No headaches.
- I hope for considerable speed improvements. Maybe also some memory savings,
at least from refactorings possible after this step.
Possible further improvements:
Read requests actually don't need to enter the processor pipeline. Instead
each server connection could get a reference to a (zxid, tree) tuple. Updates
are delivered as (zxid, newTree, triggerWatchesCallback) to the server
connections.
The watches could be managed at each server connection instead of centraly at
the DataTree.
I'm pre-announcing this now to give an explanaition of my intends with ZK.
I've created a couple of patches that refactor ZK in small incrementally and
individually tested steps as far as possible towards the immutable data tree.
The Branch is here:
https://github.com/thkoch2001/zookeeper/tree/proposed_patches
I'm working on the final step now, that really makes DataNode immutable and
which is the only larger change. I don't see yet how I can split this in
smaller steps.
Of course it's totaly up to you, whether you'd like to take this road.
Once I've got there, I'd like to do performance comparissions between old and
new. However I don't have accesses to hardware for such a test. So this would
only be possible, if somebody would like to sponsor me with access to a
cluster or some machine hours at amazon.
Best regards,
Thomas Koch, http://www.koch.ro
Re: Immutable DataTree
Posted by Ted Dunning <te...@gmail.com>.
On Tue, Nov 1, 2011 at 10:46 AM, Thomas Koch <th...@koch.ro> wrote:
> Ted Dunning:
> > We have talked about this in the past. There are several clear
> advantages.
> >
> > Do you have an appropriate immutable DataTree? What did you use to
> > implement this?
> Java? :-)
>
> I have not yet included any additional library, if that's your question.
>
> What do you mean by "appropriate"?
>
One that works, is fast and doesn't eat memory.
Did you implement your own immutable data tree? Is it lock-free? What
general method did you use?
Re: Immutable DataTree
Posted by Thomas Koch <th...@koch.ro>.
Ted Dunning:
> We have talked about this in the past. There are several clear advantages.
>
> Do you have an appropriate immutable DataTree? What did you use to
> implement this?
Java? :-)
I have not yet included any additional library, if that's your question.
What do you mean by "appropriate"?
Thomas Koch, http://www.koch.ro
Re: Immutable DataTree
Posted by Ted Dunning <te...@gmail.com>.
We have talked about this in the past. There are several clear advantages.
Do you have an appropriate immutable DataTree? What did you use to
implement this?
On Tue, Nov 1, 2011 at 9:40 AM, Thomas Koch <th...@koch.ro> wrote:
> Hi,
>
> I'm like two days away from having a version of ZooKeeper with an immutable
> DataTree on the server. This means that for any modification of the
> DataTree,
> all Nodes from the modified one up to the root are replaced by new nodes.
>
> Advantages:
>
> - No synchronisation needed when accessing the DataTree.
> - The snapshotter thread gets an immutable datatree and will write a
> consistent DataTree to the disk.
> - No headaches whether multi transactions could lead to issues with
> (de)serialization.
> - Much better testability.
> - No concurrency - No headaches.
> - I hope for considerable speed improvements. Maybe also some memory
> savings,
> at least from refactorings possible after this step.
>
> Possible further improvements:
>
> Read requests actually don't need to enter the processor pipeline. Instead
> each server connection could get a reference to a (zxid, tree) tuple.
> Updates
> are delivered as (zxid, newTree, triggerWatchesCallback) to the server
> connections.
> The watches could be managed at each server connection instead of centraly
> at
> the DataTree.
>
> I'm pre-announcing this now to give an explanaition of my intends with ZK.
>
> I've created a couple of patches that refactor ZK in small incrementally
> and
> individually tested steps as far as possible towards the immutable data
> tree.
> The Branch is here:
> https://github.com/thkoch2001/zookeeper/tree/proposed_patches
>
> I'm working on the final step now, that really makes DataNode immutable and
> which is the only larger change. I don't see yet how I can split this in
> smaller steps.
>
> Of course it's totaly up to you, whether you'd like to take this road.
>
> Once I've got there, I'd like to do performance comparissions between old
> and
> new. However I don't have accesses to hardware for such a test. So this
> would
> only be possible, if somebody would like to sponsor me with access to a
> cluster or some machine hours at amazon.
>
> Best regards,
>
> Thomas Koch, http://www.koch.ro
>
Re: Immutable DataTree
Posted by Ted Dunning <te...@gmail.com>.
Give me a budget. I will see what I can do.
On Tue, Nov 1, 2011 at 9:40 AM, Thomas Koch <th...@koch.ro> wrote:
> Once I've got there, I'd like to do performance comparissions between old
> and
> new. However I don't have accesses to hardware for such a test. So this
> would
> only be possible, if somebody would like to sponsor me with access to a
> cluster or some machine hours at amazon.
>