NIO: too many open files

[Manos Kapritsos <ma...@cs.utexas.edu>]

Hi all,

in my experiments with Zookeeper I have tried to have 1024 clients 
connected simultaneously to a single server. When that happens i get a 
"Too many open files" exception on the server side. It is my 
understanding that this is the NR_OPEN limit imposed by the kernel. 
Although it is possible to increase this limit, this is only a static 
solution, that could fail under a spike in the load.
Is there a dynamic workaround for this problem? Given that sessions are 
statically linked to the open sockets, it seems to me that there is no 
way we can manipulate the existing sockets (e.g. by closing them in a 
LRU way).

Thank you,
Manos

Re: NIO: too many open files

[Flavio Junqueira <fp...@yahoo-inc.com>]

Manos, I'm not 100% clear on what you're asking. A way to scale the  
total number of connections  in a ZooKeeper deployment is by adding  
more servers. It is true, though, that adding more servers will  
provide more connections, but still a bounded number of connections  
(changing the kernel limit or not). I think that you're suggesting  
that we should be discarding connections at some point because you  
might be concerned with an attack in which a client collapses a server  
by reaching its maximum number of connections. If this is what you're  
concerned about, then this doesn't fit into our current model. As our  
clients are not malicious by assumption, by closing a connection to a  
client we would be denying service to a legitimate client. However, I  
think it might not be too hard to enforce a total number of  
connections per IP by extending our quota mechanism.

-Flavio

On Feb 17, 2009, at 11:11 PM, Manos Kapritsos wrote:

> Hi all,
>
> in my experiments with Zookeeper I have tried to have 1024 clients  
> connected simultaneously to a single server. When that happens i get  
> a "Too many open files" exception on the server side. It is my  
> understanding that this is the NR_OPEN limit imposed by the kernel.  
> Although it is possible to increase this limit, this is only a  
> static solution, that could fail under a spike in the load.
> Is there a dynamic workaround for this problem? Given that sessions  
> are statically linked to the open sockets, it seems to me that there  
> is no way we can manipulate the existing sockets (e.g. by closing  
> them in a LRU way).
>
> Thank you,
> Manos

Re: NIO: too many open files

[Mahadev Konar <ma...@yahoo-inc.com>]

Hi Manos,
 
 The number of open files allowed for a process can be changed. 1024 is
usually a very small number. You should change that to a bigger number to
enable zookeeper to run.

Ulimit -a should show you the number of open files allowed. It can be
changed using ulimit -n.

Also, the default for ulimit is picked up from /etc/security/limits.conf (on
linux). In case you want to change that permanently for your machine.

Mahadev

On 2/17/09 2:11 PM, "Manos Kapritsos" <ma...@cs.utexas.edu> wrote:

> Hi all,
> 
> in my experiments with Zookeeper I have tried to have 1024 clients
> connected simultaneously to a single server. When that happens i get a
> "Too many open files" exception on the server side. It is my
> understanding that this is the NR_OPEN limit imposed by the kernel.
> Although it is possible to increase this limit, this is only a static
> solution, that could fail under a spike in the load.
> Is there a dynamic workaround for this problem? Given that sessions are
> statically linked to the open sockets, it seems to me that there is no
> way we can manipulate the existing sockets (e.g. by closing them in a
> LRU way).
> 
> Thank you,
> Manos