You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@curator.apache.org by Robert Kanter <rk...@cloudera.com> on 2015/01/29 18:59:15 UTC

InterProcessSemaphoreMutex Scalability

Hi all,

Is there any information on how scalable the InterProcessSemaphoreMutex
is?  I'm working on somethere where, in some cases, there could be ~600
clients trying to acquire the same lock.  Will that be a problem?
I'd go and test this directly, but I don't currently have enough machines
to try it with so I was hoping someone has already done some testing in
this area.

thanks
- Robert

Re: InterProcessSemaphoreMutex Scalability

Posted by Robert Kanter <rk...@cloudera.com>.
Great, thanks!

On Thu, Jan 29, 2015 at 1:29 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:

> The major limitation is that, by default, ZooKeeper API calls are limited
> to 1MB. The lock recipe calls getChildren() and the concatenated list of
> children can’t be more than 1MB. The node names contain the prefix “lease-“
> (6 chars), a UUID (36 chars) and the sequence number (10 chars). So, that’s
> 52 chars per lock. I don’t know if there’s other overhead. But, anyway,
> that gives you room for something like 19K lock participants. So, 600 is no
> problem. Additionally, each lock participant only watches it’s preceding
> participant so you won’t get herding as the number of clients grow.
>
> -Jordan
>
>
>
> On January 29, 2015 at 1:01:56 PM, Robert Kanter (rkanter@cloudera.com)
> wrote:
>
> Hi all,
>
> Is there any information on how scalable the InterProcessSemaphoreMutex
> is?  I'm working on somethere where, in some cases, there could be ~600
> clients trying to acquire the same lock.  Will that be a problem?
> I'd go and test this directly, but I don't currently have enough machines
> to try it with so I was hoping someone has already done some testing in
> this area.
>
> thanks
> - Robert
>
>

Re: InterProcessSemaphoreMutex Scalability

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
The major limitation is that, by default, ZooKeeper API calls are limited to 1MB. The lock recipe calls getChildren() and the concatenated list of children can’t be more than 1MB. The node names contain the prefix “lease-“ (6 chars), a UUID (36 chars) and the sequence number (10 chars). So, that’s 52 chars per lock. I don’t know if there’s other overhead. But, anyway, that gives you room for something like 19K lock participants. So, 600 is no problem. Additionally, each lock participant only watches it’s preceding participant so you won’t get herding as the number of clients grow.

-Jordan



On January 29, 2015 at 1:01:56 PM, Robert Kanter (rkanter@cloudera.com) wrote:

Hi all,

Is there any information on how scalable the InterProcessSemaphoreMutex is?  I'm working on somethere where, in some cases, there could be ~600 clients trying to acquire the same lock.  Will that be a problem?  
I'd go and test this directly, but I don't currently have enough machines to try it with so I was hoping someone has already done some testing in this area.  

thanks
- Robert