You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Dejan Markic <De...@mobik-ics.com> on 2014/12/02 23:22:38 UTC
A few ZooKeeper questions
Hello all!
Sorry for long message. But, here goes ...
We've started using ZooKeeper a few weeks ago, mostly for distributed locking. We have some performance issues, so I'll try to explain as much as possible about our configuration and applications.
For each "service", for which we need ZooKeeper, we're currently running quorum with 3 servers.
We are running version 3.3.5 on Debian 7.7. We are using MT C API using nonasync functions.
All servers have the following configuration:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/lib/zookeeper
clientPort=2181
server.1=zk_server_1:2888:3888
server.2=zk_server_2:2888:3888
server.3=zk_server_3:2888:3888
maxClientCnxns=50
skipACL=yes
On all servers, /var/lib/zookeeper directory is acctualy a 2GB tmpfs "partition".
We are running zkPurgeTxnLog.sh script every 2 minutes.
We have 6 CPU's on each server with 6GB of ram, but there's ussualy atleast one more service running on it (eg Radius).
So, let me try to explain in which way we use the zookeeper.
We have Radius server(s) receiving around 300 requests per second (authentication and accounting). We need to maintain all sessions states (start, stop update) (and sessions come and go very quickly).
NAS(es) can send many requests for same session in unordered fashion (eg, auth request comes before acct Start, and Stop comes before Update, etc).
So for each request, we request a lock from ZooKeeper. We use a bit modified recipe as found at https://zookeeper.apache.org/doc/r3.1.2/recipes.html.
The recipe we're currently using is far from perfect as we tried to implement a timeout for locking. So basically what we do is:
- create ephemeral node /SESSIONID/_xlock/lock-
- get children nodes in /SESSIONID/_xlock/
- If we are the lowest sequence, we got the lock
- If not, we wait for 10ms and call get children again untill we get the lock, or timeout expires (we ussualy wait 1 second)
After we get the lock, we write/read some stuff relevant for session in /SESSIONID/nodeName. Data is ussualy up to 20 bytes atmost. We write/read up to 3 nodes for each session.
We then insert/update session in our MySQL servers and unlock the lock in ZooKeeper.
Ussualy it takes around 30-50ms to obtain the lock from ZooKeeper. But it does happen, that timeout of 1 second occurs. It seems that sometimes ZooKeeper is performing something and is busy and response times can go way up.
Since there's a lot of nodes (/SESSIONID) left after the sessions finish, we created a script, that removes all sessions that were last modified 900 seconds ago. So this script goes through all nodes (/SESSIONID) and checks their children last modification time. If none of the children were changed in last 900 seconds, we remove the whole SESSIONID node and its children.
Questions:
- Do you think, we could go without that stupid 10ms wait in lock recipe, and just relied on zookeeper that exists() with watch will return in timely fashion? Is there little or no possibilty that one session would get stuck/deadlocked?
- If clients requesting a lock in /SESSIONID/_xlock/lock- connect to different servers in quorum, is there a need to perform a sync before checking if we're the node with minimal sequence number?
- Would adding more servers to quorum increase the performance time? Would performance be more constant that way?
- Are we performing too many writes to zookeeper?
- If we have 300k child nodes (eg 300k /SESSIONID nodes) would that be a performace issue?
- When we make the lock path, we use recursive create, which basically does a create() for each node, and if ZNODEEXISTS is returned, we simply go on. I see a *lot* of these messages in log:
[ProcessThread:-1:PrepRequestProcessor@419] - Got user-level KeeperException when processing sessionid:0x34a0cbf41360018 type:create cxid:0x548a2589 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/SESSIONID/_xlock Error:KeeperErrorCode = NodeExists for /SESSIONID/_xlock
Would it be better if exists() would be called for each before creating if needed? And, do we need to sync() before checking with exists()?
- Would upgrading to newer version improve performance?
Any input would be greatly appreciated!
Kind regards,
Dejan Markic
Re: A few ZooKeeper questions
Posted by Maciej Smoleński <je...@gmail.com>.
Hello,
I answered the questions best to my knowledge.
Please note that I only have theoretical knowledge about ZooKeeper, but no
practical experience.
I'm not sure if ZooKeeper can handle such a big number of operations that
you perform, Your load type seems that there are many creates/writes
(compared to reads), and the performance might be a problem because of so
many creates/writes. How many creates/writes do you want to do per second ?
Do you think, we could go without that stupid 10ms wait in lock recipe, and
just relied on zookeeper that exists() with watch will return in timely
fashion? Is there little or no possibilty that one session would get
stuck/deadlocked?
I think You should use watches if You can rather than checking state every
10ms. Checking state every 10ms is not good for performance. I think with
watches the load will be lower.
Note that if the server the client is connected to is not sync with the
leader then the notification will come when it is synced with the leader
(see syncLimit parameter) - I don't expect that big delays should be common.
If clients requesting a lock in /SESSIONID/_xlock/lock- connect to
different servers in quorum, is there a need to perform a sync before
checking if we're the node with minimal sequence number?
Sync could be eliminated here. If the client switches to a new server, it
is ensured that the client will switch to a server with state at least as
fresh as state on previous server. If client uses watches he will get
notifications (he was registered for) of any changed made in between. So
You shouldn't miss anything during server switch (session stays valid among
switches). The new server might also not have the newest state, but will be
synced with leader after some time (syncLimit parameter) or client session
will detect long delay, and switch to another server.
Would adding more servers to quorum increase the performance time? Would
performance be more constant that way?
It depends on your operation type, for 100% writes the best performance is
with lowest number of zk servers, for 100% reads when you are fine with
stale data, and not issue sync requests, than the more zk servers you have
the performance will be better.
When we make the lock path, we use recursive create. Would it be better if
exists() would be called for each before creating if needed? And, do we
need to sync() before checking with exists()?
I think it should reduce the load. I think that create always requires to
be processed by leader and quorum and should be avoided if possible, Exists
could be handled locally on single server, but the result could be stale.
If you expect that exists for some path will not change very often, than it
seams reasonable to use exists without sync first, and wait for
notifications otherwise. I think You should avoid sync if you can.
Kind regards,
Maciej Smoleński
On Tue, Dec 2, 2014 at 11:22 PM, Dejan Markic <De...@mobik-ics.com>
wrote:
> Hello all!
>
> Sorry for long message. But, here goes ...
>
> We've started using ZooKeeper a few weeks ago, mostly for distributed
> locking. We have some performance issues, so I'll try to explain as much as
> possible about our configuration and applications.
>
> For each "service", for which we need ZooKeeper, we're currently running
> quorum with 3 servers.
> We are running version 3.3.5 on Debian 7.7. We are using MT C API using
> nonasync functions.
> All servers have the following configuration:
> tickTime=2000
> initLimit=10
> syncLimit=5
> dataDir=/var/lib/zookeeper
> clientPort=2181
> server.1=zk_server_1:2888:3888
> server.2=zk_server_2:2888:3888
> server.3=zk_server_3:2888:3888
> maxClientCnxns=50
> skipACL=yes
>
> On all servers, /var/lib/zookeeper directory is acctualy a 2GB tmpfs
> "partition".
> We are running zkPurgeTxnLog.sh script every 2 minutes.
> We have 6 CPU's on each server with 6GB of ram, but there's ussualy
> atleast one more service running on it (eg Radius).
>
> So, let me try to explain in which way we use the zookeeper.
> We have Radius server(s) receiving around 300 requests per second
> (authentication and accounting). We need to maintain all sessions states
> (start, stop update) (and sessions come and go very quickly).
> NAS(es) can send many requests for same session in unordered fashion (eg,
> auth request comes before acct Start, and Stop comes before Update, etc).
> So for each request, we request a lock from ZooKeeper. We use a bit
> modified recipe as found at
> https://zookeeper.apache.org/doc/r3.1.2/recipes.html.
> The recipe we're currently using is far from perfect as we tried to
> implement a timeout for locking. So basically what we do is:
>
> - create ephemeral node /SESSIONID/_xlock/lock-
> - get children nodes in /SESSIONID/_xlock/
> - If we are the lowest sequence, we got the lock
> - If not, we wait for 10ms and call get children again untill we get the
> lock, or timeout expires (we ussualy wait 1 second)
>
> After we get the lock, we write/read some stuff relevant for session in
> /SESSIONID/nodeName. Data is ussualy up to 20 bytes atmost. We write/read
> up to 3 nodes for each session.
> We then insert/update session in our MySQL servers and unlock the lock in
> ZooKeeper.
>
> Ussualy it takes around 30-50ms to obtain the lock from ZooKeeper. But it
> does happen, that timeout of 1 second occurs. It seems that sometimes
> ZooKeeper is performing something and is busy and response times can go way
> up.
>
> Since there's a lot of nodes (/SESSIONID) left after the sessions finish,
> we created a script, that removes all sessions that were last modified 900
> seconds ago. So this script goes through all nodes (/SESSIONID) and checks
> their children last modification time. If none of the children were changed
> in last 900 seconds, we remove the whole SESSIONID node and its children.
>
> Questions:
> - Do you think, we could go without that stupid 10ms wait in lock recipe,
> and just relied on zookeeper that exists() with watch will return in timely
> fashion? Is there little or no possibilty that one session would get
> stuck/deadlocked?
> - If clients requesting a lock in /SESSIONID/_xlock/lock- connect to
> different servers in quorum, is there a need to perform a sync before
> checking if we're the node with minimal sequence number?
> - Would adding more servers to quorum increase the performance time? Would
> performance be more constant that way?
> - Are we performing too many writes to zookeeper?
> - If we have 300k child nodes (eg 300k /SESSIONID nodes) would that be a
> performace issue?
> - When we make the lock path, we use recursive create, which basically
> does a create() for each node, and if ZNODEEXISTS is returned, we simply go
> on. I see a *lot* of these messages in log:
> [ProcessThread:-1:PrepRequestProcessor@419] - Got user-level
> KeeperException when processing sessionid:0x34a0cbf41360018 type:create
> cxid:0x548a2589 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/SESSIONID/_xlock Error:KeeperErrorCode = NodeExists for
> /SESSIONID/_xlock
> Would it be better if exists() would be called for each before creating
> if needed? And, do we need to sync() before checking with exists()?
> - Would upgrading to newer version improve performance?
>
> Any input would be greatly appreciated!
>
> Kind regards,
> Dejan Markic
>
>
>
>
>
RE: A few ZooKeeper questions
Posted by Rakesh R <ra...@huawei.com>.
Hi,
>>>>>>>- Do you think, we could go without that stupid 10ms wait in lock recipe, and just relied on zookeeper that exists() with watch will return in timely fashion? Is there little or no possibilty that one session would get stuck/deadlocked?
Since you are using EPHEMRAL_SEQUENCE node and the lowest one will get always lock. I suggest rather than doing 10secs wait. You can have a watcher implementation which looks only my predecessor lock znode deletion. Also, on reconnection (in disconnected and then syncconnected) you can getChildren and see my predecessor still exists or not.
>>>>>>>>- If clients requesting a lock in /SESSIONID/_xlock/lock- connect to different servers in quorum, is there a need to perform a sync before checking if we're the node with minimal sequence number?
Not required, since you are using EHPEMRAL_SEQUENCE all the writes will go through the Leader and if my client lock is "lock_000010" it means Leader has created "lock_000009" before me. getChildren() will return [lock_000009, lock_000010]
>>>>>>>- Would adding more servers to quorum increase the performance time? Would performance be more constant that way?
If you increase the number of servers it will affect the write latency and will affect the write performance. If you want to improve the read performance you can add "Observers".
Please refer: http://zookeeper.apache.org/doc/trunk/zookeeperObservers.html
>>>>>>- Are we performing too many writes to zookeeper?
>>>>>>>- If we have 300k child nodes (eg 300k /SESSIONID nodes) would that be a performace issue?
If there are many writes per secs concurrently, there could be cases of bottle neck at the disk level or network packets. This will affect the performance. But you can do benchmark in your env.
>From the above description what I understood is you are creating and deleting the znodes very frequently. In that case, there will be more snapshot and transaction logs in dataDir/dataLogDir respectively. Please ensure the purge tasks is enabled and do the purging periodically.
>>>>>>>- When we make the lock path, we use recursive create, which basically does a create() for each node, and if ZNODEEXISTS is returned, we simply go on. I see a *lot* of these messages in log:
[ProcessThread:-1:PrepRequestProcessor@419] - Got user-level KeeperException when processing sessionid:0x34a0cbf41360018 type:create cxid:0x548a2589 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/SESSIONID/_xlock Error:KeeperErrorCode = NodeExists for /SESSIONID/_xlock Would it be better if exists() would be called for each before creating if needed? And, do we need to sync() before checking with exists()?
exists() api call will reduce the exceptions, but this can't be avoided I feel. Because you have many concurrent writes per second. Sometimes exists() may not seen the concurrently created /SESSIONID/_xlock znode.
- Would upgrading to newer version improve performance?
3.3.5 version is quite old, please take the stable 3.4.* release.
Cheers,
Rakesh
-----Original Message-----
From: Dejan Markic [mailto:Dejan.Markic@mobik-ics.com]
Sent: 03 December 2014 20:32
To: user@zookeeper.apache.org
Subject: FW: A few ZooKeeper questions
I'm sending this again, as I don't know if I sent it to the correct e-mail.
Kind regards,
Dejan
-----Original Message-----
From: Dejan Markic
Sent: Tuesday, December 2, 2014 11:23 PM
To: zookeeper-user@hadoop.apache.org
Subject: A few ZooKeeper questions
Hello all!
Sorry for long message. But, here goes ...
We've started using ZooKeeper a few weeks ago, mostly for distributed locking. We have some performance issues, so I'll try to explain as much as possible about our configuration and applications.
For each "service", for which we need ZooKeeper, we're currently running quorum with 3 servers.
We are running version 3.3.5 on Debian 7.7. We are using MT C API using nonasync functions.
All servers have the following configuration:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/lib/zookeeper
clientPort=2181
server.1=zk_server_1:2888:3888
server.2=zk_server_2:2888:3888
server.3=zk_server_3:2888:3888
maxClientCnxns=50
skipACL=yes
On all servers, /var/lib/zookeeper directory is acctualy a 2GB tmpfs "partition".
We are running zkPurgeTxnLog.sh script every 2 minutes.
We have 6 CPU's on each server with 6GB of ram, but there's ussualy atleast one more service running on it (eg Radius).
So, let me try to explain in which way we use the zookeeper.
We have Radius server(s) receiving around 300 requests per second (authentication and accounting). We need to maintain all sessions states (start, stop update) (and sessions come and go very quickly).
NAS(es) can send many requests for same session in unordered fashion (eg, auth request comes before acct Start, and Stop comes before Update, etc).
So for each request, we request a lock from ZooKeeper. We use a bit modified recipe as found at https://zookeeper.apache.org/doc/r3.1.2/recipes.html.
The recipe we're currently using is far from perfect as we tried to implement a timeout for locking. So basically what we do is:
- create ephemeral node /SESSIONID/_xlock/lock-
- get children nodes in /SESSIONID/_xlock/
- If we are the lowest sequence, we got the lock
- If not, we wait for 10ms and call get children again untill we get the lock, or timeout expires (we ussualy wait 1 second)
After we get the lock, we write/read some stuff relevant for session in /SESSIONID/nodeName. Data is ussualy up to 20 bytes atmost. We write/read up to 3 nodes for each session.
We then insert/update session in our MySQL servers and unlock the lock in ZooKeeper.
Ussualy it takes around 30-50ms to obtain the lock from ZooKeeper. But it does happen, that timeout of 1 second occurs. It seems that sometimes ZooKeeper is performing something and is busy and response times can go way up.
Since there's a lot of nodes (/SESSIONID) left after the sessions finish, we created a script, that removes all sessions that were last modified 900 seconds ago. So this script goes through all nodes (/SESSIONID) and checks their children last modification time. If none of the children were changed in last 900 seconds, we remove the whole SESSIONID node and its children.
Questions:
- Do you think, we could go without that stupid 10ms wait in lock recipe, and just relied on zookeeper that exists() with watch will return in timely fashion? Is there little or no possibilty that one session would get stuck/deadlocked?
- If clients requesting a lock in /SESSIONID/_xlock/lock- connect to different servers in quorum, is there a need to perform a sync before checking if we're the node with minimal sequence number?
- Would adding more servers to quorum increase the performance time? Would performance be more constant that way?
- Are we performing too many writes to zookeeper?
- If we have 300k child nodes (eg 300k /SESSIONID nodes) would that be a performace issue?
- When we make the lock path, we use recursive create, which basically does a create() for each node, and if ZNODEEXISTS is returned, we simply go on. I see a *lot* of these messages in log:
[ProcessThread:-1:PrepRequestProcessor@419] - Got user-level KeeperException when processing sessionid:0x34a0cbf41360018 type:create cxid:0x548a2589 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/SESSIONID/_xlock Error:KeeperErrorCode = NodeExists for /SESSIONID/_xlock Would it be better if exists() would be called for each before creating if needed? And, do we need to sync() before checking with exists()?
- Would upgrading to newer version improve performance?
Any input would be greatly appreciated!
Kind regards,
Dejan Markic
FW: A few ZooKeeper questions
Posted by Dejan Markic <De...@mobik-ics.com>.
I'm sending this again, as I don't know if I sent it to the correct e-mail.
Kind regards,
Dejan
-----Original Message-----
From: Dejan Markic
Sent: Tuesday, December 2, 2014 11:23 PM
To: zookeeper-user@hadoop.apache.org
Subject: A few ZooKeeper questions
Hello all!
Sorry for long message. But, here goes ...
We've started using ZooKeeper a few weeks ago, mostly for distributed locking. We have some performance issues, so I'll try to explain as much as possible about our configuration and applications.
For each "service", for which we need ZooKeeper, we're currently running quorum with 3 servers.
We are running version 3.3.5 on Debian 7.7. We are using MT C API using nonasync functions.
All servers have the following configuration:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/lib/zookeeper
clientPort=2181
server.1=zk_server_1:2888:3888
server.2=zk_server_2:2888:3888
server.3=zk_server_3:2888:3888
maxClientCnxns=50
skipACL=yes
On all servers, /var/lib/zookeeper directory is acctualy a 2GB tmpfs "partition".
We are running zkPurgeTxnLog.sh script every 2 minutes.
We have 6 CPU's on each server with 6GB of ram, but there's ussualy atleast one more service running on it (eg Radius).
So, let me try to explain in which way we use the zookeeper.
We have Radius server(s) receiving around 300 requests per second (authentication and accounting). We need to maintain all sessions states (start, stop update) (and sessions come and go very quickly).
NAS(es) can send many requests for same session in unordered fashion (eg, auth request comes before acct Start, and Stop comes before Update, etc).
So for each request, we request a lock from ZooKeeper. We use a bit modified recipe as found at https://zookeeper.apache.org/doc/r3.1.2/recipes.html.
The recipe we're currently using is far from perfect as we tried to implement a timeout for locking. So basically what we do is:
- create ephemeral node /SESSIONID/_xlock/lock-
- get children nodes in /SESSIONID/_xlock/
- If we are the lowest sequence, we got the lock
- If not, we wait for 10ms and call get children again untill we get the lock, or timeout expires (we ussualy wait 1 second)
After we get the lock, we write/read some stuff relevant for session in /SESSIONID/nodeName. Data is ussualy up to 20 bytes atmost. We write/read up to 3 nodes for each session.
We then insert/update session in our MySQL servers and unlock the lock in ZooKeeper.
Ussualy it takes around 30-50ms to obtain the lock from ZooKeeper. But it does happen, that timeout of 1 second occurs. It seems that sometimes ZooKeeper is performing something and is busy and response times can go way up.
Since there's a lot of nodes (/SESSIONID) left after the sessions finish, we created a script, that removes all sessions that were last modified 900 seconds ago. So this script goes through all nodes (/SESSIONID) and checks their children last modification time. If none of the children were changed in last 900 seconds, we remove the whole SESSIONID node and its children.
Questions:
- Do you think, we could go without that stupid 10ms wait in lock recipe, and just relied on zookeeper that exists() with watch will return in timely fashion? Is there little or no possibilty that one session would get stuck/deadlocked?
- If clients requesting a lock in /SESSIONID/_xlock/lock- connect to different servers in quorum, is there a need to perform a sync before checking if we're the node with minimal sequence number?
- Would adding more servers to quorum increase the performance time? Would performance be more constant that way?
- Are we performing too many writes to zookeeper?
- If we have 300k child nodes (eg 300k /SESSIONID nodes) would that be a performace issue?
- When we make the lock path, we use recursive create, which basically does a create() for each node, and if ZNODEEXISTS is returned, we simply go on. I see a *lot* of these messages in log:
[ProcessThread:-1:PrepRequestProcessor@419] - Got user-level KeeperException when processing sessionid:0x34a0cbf41360018 type:create cxid:0x548a2589 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/SESSIONID/_xlock Error:KeeperErrorCode = NodeExists for /SESSIONID/_xlock Would it be better if exists() would be called for each before creating if needed? And, do we need to sync() before checking with exists()?
- Would upgrading to newer version improve performance?
Any input would be greatly appreciated!
Kind regards,
Dejan Markic