You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Baskar Duraikannu <ba...@outlook.com> on 2013/07/31 13:16:28 UTC
Zookeeper performance
Hello
We are looking to use zookeeper for optimistic concurrency. Basically when the user saves data on a screen, we need to lock, read to ensure that no one else has changed the row while user is editing data, persist data and unlock znode.
If the app/thread does not get a lock, we may set a watch so that polling is avoided.
Our application is write intensive certain times of the day. We may get about 100k requests per second. Can zookeeper handle this volume?
RE: Zookeeper performance
Posted by Baskar Duraikannu <ba...@outlook.com>.
We are using optimistic concurrency similar to one explained in http://en.wikipedia.org/wiki/Optimistic_concurrency_control
I went through the "version" concept in zookeeper. I do not understand how that could be used for "concurrency control". Are you suggesting that all writes go to zookeeper first before making it to the database.
We are trying to do it differently... For example, let us say we are trying to update rowKey 1 of Person table,
1. Create a ephemeral node /locks/Person/rowkey12. Zookeeper may create a node called /locks/Person/rowKey1-0000 (This means that we got lock)3. If we got, /locks/Person/rowKey1-0001 , we decrement and keep checking child nodes of /locks/Person until /locks/Person/rowKey1-0001 becomes the first one in the list.
Pls help/
> From: phunt@apache.org
> Date: Wed, 31 Jul 2013 08:47:01 -0700
> Subject: Re: Zookeeper performance
> To: user@zookeeper.apache.org
>
> On Wed, Jul 31, 2013 at 4:16 AM, Baskar Duraikannu
> <ba...@outlook.com> wrote:
> >
> > We are looking to use zookeeper for optimistic concurrency. Basically when the user saves data on a screen, we need to lock, read to ensure that no one else has changed the row while user is editing data, persist data and unlock znode.
> >
>
> Why not use the built in optimistic locking that ZK provides? See the
> "version" parameter here:
> http://zookeeper.apache.org/doc/r3.4.5/api/org/apache/zookeeper/ZooKeeper.html#getData(java.lang.String,
> boolean, org.apache.zookeeper.AsyncCallback.DataCallback,
> java.lang.Object)
> http://zookeeper.apache.org/doc/r3.4.5/api/org/apache/zookeeper/ZooKeeper.html#setData(java.lang.String,
> byte[], int, org.apache.zookeeper.AsyncCallback.StatCallback,
> java.lang.Object)
>
> > If the app/thread does not get a lock, we may set a watch so that polling is avoided.
> >
> > Our application is write intensive certain times of the day. We may get about 100k requests per second. Can zookeeper handle this volume?
>
> Unlikely you'll see this level of performance:
> http://zookeeper.apache.org/doc/r3.4.5/zookeeperOver.html#Performance
>
> Patrick
Re: Zookeeper performance
Posted by Patrick Hunt <ph...@apache.org>.
On Wed, Jul 31, 2013 at 4:16 AM, Baskar Duraikannu
<ba...@outlook.com> wrote:
>
> We are looking to use zookeeper for optimistic concurrency. Basically when the user saves data on a screen, we need to lock, read to ensure that no one else has changed the row while user is editing data, persist data and unlock znode.
>
Why not use the built in optimistic locking that ZK provides? See the
"version" parameter here:
http://zookeeper.apache.org/doc/r3.4.5/api/org/apache/zookeeper/ZooKeeper.html#getData(java.lang.String,
boolean, org.apache.zookeeper.AsyncCallback.DataCallback,
java.lang.Object)
http://zookeeper.apache.org/doc/r3.4.5/api/org/apache/zookeeper/ZooKeeper.html#setData(java.lang.String,
byte[], int, org.apache.zookeeper.AsyncCallback.StatCallback,
java.lang.Object)
> If the app/thread does not get a lock, we may set a watch so that polling is avoided.
>
> Our application is write intensive certain times of the day. We may get about 100k requests per second. Can zookeeper handle this volume?
Unlikely you'll see this level of performance:
http://zookeeper.apache.org/doc/r3.4.5/zookeeperOver.html#Performance
Patrick
RE: Zookeeper performance
Posted by Baskar Duraikannu <ba...@outlook.com>.
Thank you.
> Date: Wed, 31 Jul 2013 20:09:59 +0100
> From: bill@dehora.net
> To: user@zookeeper.apache.org
> Subject: Re: Zookeeper performance
>
>
> > We may get about 100k requests per second. Can zookeeper handle this volume?
>
> Depends on your r/w ratio. ZK isn't always ideal for 'user-scale' write heavy workload (lots of things being tracked and updated) . My experience is that you'll have memory pressure (due to the size of the tree) and disk pressure (due to the frequency of snapshotting). You'll need to look at ssds or ram disks for the latter and expect to invest time tuning the JVM, ZK parameters for the former.
>
> I agree that it is likely to be error prone without a lot of careful engineering; a design that avoids/reduces locking or a technology that has locking built-in might be a better fit.
>
> Bill
>
>
>
>
> On Wednesday 31 July 2013 at 12:16, Baskar Duraikannu wrote:
>
> > Hello
> >
> > We are looking to use zookeeper for optimistic concurrency. Basically when the user saves data on a screen, we need to lock, read to ensure that no one else has changed the row while user is editing data, persist data and unlock znode.
> >
> > If the app/thread does not get a lock, we may set a watch so that polling is avoided.
> >
> > Our application is write intensive certain times of the day. We may get about 100k requests per second. Can zookeeper handle this volume?
>
>
Re: Zookeeper performance
Posted by Bill de hÓra <bi...@dehora.net>.
> We may get about 100k requests per second. Can zookeeper handle this volume?
Depends on your r/w ratio. ZK isn't always ideal for 'user-scale' write heavy workload (lots of things being tracked and updated) . My experience is that you'll have memory pressure (due to the size of the tree) and disk pressure (due to the frequency of snapshotting). You'll need to look at ssds or ram disks for the latter and expect to invest time tuning the JVM, ZK parameters for the former.
I agree that it is likely to be error prone without a lot of careful engineering; a design that avoids/reduces locking or a technology that has locking built-in might be a better fit.
Bill
On Wednesday 31 July 2013 at 12:16, Baskar Duraikannu wrote:
> Hello
>
> We are looking to use zookeeper for optimistic concurrency. Basically when the user saves data on a screen, we need to lock, read to ensure that no one else has changed the row while user is editing data, persist data and unlock znode.
>
> If the app/thread does not get a lock, we may set a watch so that polling is avoided.
>
> Our application is write intensive certain times of the day. We may get about 100k requests per second. Can zookeeper handle this volume?
Re: Zookeeper performance
Posted by Ted Dunning <te...@gmail.com>.
Generally, ZK is much better as a coordination layer.
Starting with an expected transaction load well above the normal limits of
operation is not a grand idea.
Much better to do something simpler like have ZK coordinate shard masters
that each use conventional methods for handling transactions (see voltdb
for one approach to sharding well to allow each transaction to never span
shards).
Similarly, you can also shard and maintain version numbers, transaction
id's and an in-memory transaction table. This allows multi-shard MVCC
commit semantics but can be a bit tricky to deal with transactions stalled
by dead nodes.
Using ZK for the raw transaction stream isn't a grand idea, however.
On Wed, Jul 31, 2013 at 4:05 PM, Henry Robinson <he...@cloudera.com> wrote:
> So how about the following optimistic approach:
>
> 1. Read the current version of the database (stored in a znode's version
> metadata). If it is even, wait and try again; even numbers mean someone is
> committing and the DB might be in an inconsistent state. Then read the
> state from the database your update will rely upon (user1.name, in this
> instance). You must also be able to atomically read the current version
> from the database as well as zookeeper, to ensure that the data is from the
> version you think it is. If the DB version does not match the ZK version,
> restart.
> 2. Once an update is ready to commit, test-and-increment the current
> version in ZK to an even number, write your update to the DB, along with
> the eventual version of the data (the next odd number).
> 3. Increment the current version in ZK to an odd number.
>
> The even / odd distinction means that you can detect when someone else is
> updating the database, since otherwise there's no way to do so atomically
> with an update to ZK (so another transaction can't tell if you've finished
> your update or not, and so doesn't know when to wait until).
>
> The problem is failure - what happens if a client fails while it's writing
> a transaction? Eventually someone can increment the transaction number, and
> if you provide an 'undo' log before you make any changes, that client can
> possibly recover from a partial commit. But at this point you need to
> understand your application's requirements in much more detail than we do
> to make recommendations.
>
> In particular, your storage layer may offer sufficiently powerful
> primitives such that you don't need ZK; although if it's a filesystem then
> that probably isn't true.
>
> Henry
>
>
> On 31 July 2013 15:51, Baskar Duraikannu <baskar.duraikannu@outlook.com
> >wrote:
>
> > We cannot always resolve conflicts ourselves. For example, let us say
> that
> > a) user1 changed the name from 'Kathy' to Katherineb) user2 changes the
> > name from 'Kathy' to 'Kat'
> > Both read 'Kathy' as input; user1's update succeeded. If we need to let
> > user2 know that something has changed as this may result in the user not
> > changing 'Kathy' to 'Kat' (as an example).
> > Hope this explains
> >
> > > Date: Wed, 31 Jul 2013 07:49:39 -0400
> > > Subject: Re: Zookeeper performance
> > > From: camille@apache.org
> > > To: user@zookeeper.apache.org
> > >
> > > This sounds highly error prone to me regardless of whether or not
> > zookeeper
> > > can handle the load-. Why not just use a standard transaction model
> with
> > a
> > > vector clock or other timing device to detect conflicts so you don't
> have
> > > to worry about a second server to talk to (zookeeper) to do an update?
> > > On Jul 31, 2013 7:17 AM, "Baskar Duraikannu" <
> > baskar.duraikannu@outlook.com>
> > > wrote:
> > >
> > > > Hello
> > > >
> > > > We are looking to use zookeeper for optimistic concurrency. Basically
> > when
> > > > the user saves data on a screen, we need to lock, read to ensure
> that
> > no
> > > > one else has changed the row while user is editing data, persist data
> > and
> > > > unlock znode.
> > > >
> > > > If the app/thread does not get a lock, we may set a watch so that
> > polling
> > > > is avoided.
> > > >
> > > > Our application is write intensive certain times of the day. We may
> get
> > > > about 100k requests per second. Can zookeeper handle this volume?
> >
> >
>
>
>
> --
> Henry Robinson
> Software Engineer
> Cloudera
> 415-994-6679
>
Re: Zookeeper performance
Posted by Henry Robinson <he...@cloudera.com>.
So how about the following optimistic approach:
1. Read the current version of the database (stored in a znode's version
metadata). If it is even, wait and try again; even numbers mean someone is
committing and the DB might be in an inconsistent state. Then read the
state from the database your update will rely upon (user1.name, in this
instance). You must also be able to atomically read the current version
from the database as well as zookeeper, to ensure that the data is from the
version you think it is. If the DB version does not match the ZK version,
restart.
2. Once an update is ready to commit, test-and-increment the current
version in ZK to an even number, write your update to the DB, along with
the eventual version of the data (the next odd number).
3. Increment the current version in ZK to an odd number.
The even / odd distinction means that you can detect when someone else is
updating the database, since otherwise there's no way to do so atomically
with an update to ZK (so another transaction can't tell if you've finished
your update or not, and so doesn't know when to wait until).
The problem is failure - what happens if a client fails while it's writing
a transaction? Eventually someone can increment the transaction number, and
if you provide an 'undo' log before you make any changes, that client can
possibly recover from a partial commit. But at this point you need to
understand your application's requirements in much more detail than we do
to make recommendations.
In particular, your storage layer may offer sufficiently powerful
primitives such that you don't need ZK; although if it's a filesystem then
that probably isn't true.
Henry
On 31 July 2013 15:51, Baskar Duraikannu <ba...@outlook.com>wrote:
> We cannot always resolve conflicts ourselves. For example, let us say that
> a) user1 changed the name from 'Kathy' to Katherineb) user2 changes the
> name from 'Kathy' to 'Kat'
> Both read 'Kathy' as input; user1's update succeeded. If we need to let
> user2 know that something has changed as this may result in the user not
> changing 'Kathy' to 'Kat' (as an example).
> Hope this explains
>
> > Date: Wed, 31 Jul 2013 07:49:39 -0400
> > Subject: Re: Zookeeper performance
> > From: camille@apache.org
> > To: user@zookeeper.apache.org
> >
> > This sounds highly error prone to me regardless of whether or not
> zookeeper
> > can handle the load-. Why not just use a standard transaction model with
> a
> > vector clock or other timing device to detect conflicts so you don't have
> > to worry about a second server to talk to (zookeeper) to do an update?
> > On Jul 31, 2013 7:17 AM, "Baskar Duraikannu" <
> baskar.duraikannu@outlook.com>
> > wrote:
> >
> > > Hello
> > >
> > > We are looking to use zookeeper for optimistic concurrency. Basically
> when
> > > the user saves data on a screen, we need to lock, read to ensure that
> no
> > > one else has changed the row while user is editing data, persist data
> and
> > > unlock znode.
> > >
> > > If the app/thread does not get a lock, we may set a watch so that
> polling
> > > is avoided.
> > >
> > > Our application is write intensive certain times of the day. We may get
> > > about 100k requests per second. Can zookeeper handle this volume?
>
>
--
Henry Robinson
Software Engineer
Cloudera
415-994-6679
RE: Zookeeper performance
Posted by Baskar Duraikannu <ba...@outlook.com>.
We cannot always resolve conflicts ourselves. For example, let us say that
a) user1 changed the name from 'Kathy' to Katherineb) user2 changes the name from 'Kathy' to 'Kat'
Both read 'Kathy' as input; user1's update succeeded. If we need to let user2 know that something has changed as this may result in the user not changing 'Kathy' to 'Kat' (as an example).
Hope this explains
> Date: Wed, 31 Jul 2013 07:49:39 -0400
> Subject: Re: Zookeeper performance
> From: camille@apache.org
> To: user@zookeeper.apache.org
>
> This sounds highly error prone to me regardless of whether or not zookeeper
> can handle the load-. Why not just use a standard transaction model with a
> vector clock or other timing device to detect conflicts so you don't have
> to worry about a second server to talk to (zookeeper) to do an update?
> On Jul 31, 2013 7:17 AM, "Baskar Duraikannu" <ba...@outlook.com>
> wrote:
>
> > Hello
> >
> > We are looking to use zookeeper for optimistic concurrency. Basically when
> > the user saves data on a screen, we need to lock, read to ensure that no
> > one else has changed the row while user is editing data, persist data and
> > unlock znode.
> >
> > If the app/thread does not get a lock, we may set a watch so that polling
> > is avoided.
> >
> > Our application is write intensive certain times of the day. We may get
> > about 100k requests per second. Can zookeeper handle this volume?
Re: Zookeeper performance
Posted by Camille Fournier <ca...@apache.org>.
This sounds highly error prone to me regardless of whether or not zookeeper
can handle the load-. Why not just use a standard transaction model with a
vector clock or other timing device to detect conflicts so you don't have
to worry about a second server to talk to (zookeeper) to do an update?
On Jul 31, 2013 7:17 AM, "Baskar Duraikannu" <ba...@outlook.com>
wrote:
> Hello
>
> We are looking to use zookeeper for optimistic concurrency. Basically when
> the user saves data on a screen, we need to lock, read to ensure that no
> one else has changed the row while user is editing data, persist data and
> unlock znode.
>
> If the app/thread does not get a lock, we may set a watch so that polling
> is avoided.
>
> Our application is write intensive certain times of the day. We may get
> about 100k requests per second. Can zookeeper handle this volume?