You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Zhemzhitsky Sergey <Se...@troika.ru> on 2011/08/01 12:53:09 UTC
Too many open files
Hi there,
We have already set ulimit to be able to open 16384 files, but the error still happens from time to time.
Could you help us to identify the possible sources of this issue.
Best Regards,
Sergey Zhemzhitsky
Information Technology Division
Troika Dialog, 4, Romanov lane, Moscow 125009, Russia
Phone. +7 495 2580500 ext. 1246
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
RE: Too many open files
Posted by Zhemzhitsky Sergey <Se...@troika.ru>.
Gordon, thanks a lot.
We expect to have no more than 2 connections for each queue, so it's equal to 100 connections.
Our queues are static, so such a behavior seems to be a handle leak.
Best Regards,
Sergey Zhemzhitsky
Information Technology Division
Troika Dialog, 4, Romanov lane, Moscow 125009, Russia
Phone. +7 495 2580500 ext. 1246
-----Original Message-----
From: Gordon Sim [mailto:gsim@redhat.com]
Sent: Tuesday, August 02, 2011 2:52 PM
To: users@qpid.apache.org
Subject: Re: Too many open files
On 08/02/2011 11:34 AM, Zhemzhitsky Sergey wrote:
> So we have 20 queues with --file-count=64 option and 25 queues with --file-count=8 option.
>
> According to the formula nfh=nq * (nf + 1) to estimate the number of
> file handles from the following link
> http://www.redhat.com/f/pdf/Large_Journal.pdf
>
> nfh=20 * (64 + 1) + 25 * (8 + 1) = 1525
>
> But the maximum number of the open files for qpid is set to 16384 that is ten times greater.
What about connections (qpid-stat -c), how many of those do you expect to have?
Are the queues fairly static (i.e. the queues themselves don't really change, just the messages on them)? I guess it is also possible there is some sort of leak of handles(?)...
If you could monitor the open file handles during operation (or testing) and observe any patterns that may also help.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Too many open files
Posted by Gordon Sim <gs...@redhat.com>.
On 08/02/2011 11:34 AM, Zhemzhitsky Sergey wrote:
> So we have 20 queues with --file-count=64 option and 25 queues with --file-count=8 option.
>
> According to the formula nfh=nq * (nf + 1) to estimate the number of file handles from the following link http://www.redhat.com/f/pdf/Large_Journal.pdf
>
> nfh=20 * (64 + 1) + 25 * (8 + 1) = 1525
>
> But the maximum number of the open files for qpid is set to 16384 that is ten times greater.
What about connections (qpid-stat -c), how many of those do you expect
to have?
Are the queues fairly static (i.e. the queues themselves don't really
change, just the messages on them)? I guess it is also possible there is
some sort of leak of handles(?)...
If you could monitor the open file handles during operation (or testing)
and observe any patterns that may also help.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: Too many open files
Posted by Zhemzhitsky Sergey <Se...@troika.ru>.
Hi Gordon,
Here is the output of the qpid-config queues command
Queue Name Attributes
======================================================================
Queue1 --durable --file-size=128 --file-count=64
Queue2 --durable --file-size=128 --file-count=64
Queue3 --durable --file-size=128 --file-count=64
Queue4 --durable
Queue5 --durable --file-size=128 --file-count=64
Queue6 --durable
Queue7 --durable
Queue8 --durable --file-size=128 --file-count=64
Queue9 --durable
Queue10 --durable
Queue11 --durable --file-size=128 --file-count=64
Queue12 --durable --file-size=128 --file-count=64
Queue13 --durable
Queue14 --durable
Queue15 --durable
Queue16 --durable --file-size=24 --file-count=8
Queue17 --durable --file-size=24 --file-count=8
Queue18 --durable
Queue19 --durable
Queue20 --durable --file-size=128 --file-count=64
Queue21 --durable --file-size=128 --file-count=64
Queue22 --durable
Queue23 --durable
Queue24 --durable
Queue25 --durable
Queue26 --durable
Queue27 --durable
Queue28 --durable --cluster-durable --file-size=24 --file-count=8
Queue29 --durable --cluster-durable --file-size=128 --file-count=64
Queue30 --durable --cluster-durable --file-size=128 --file-count=64
Queue31 --durable --cluster-durable --file-size=24 --file-count=8
Queue32 --durable --cluster-durable --file-size=128 --file-count=64
Queue33 --durable --cluster-durable --file-size=128 --file-count=64
Queue34 --durable --cluster-durable --file-size=24 --file-count=8
Queue35 --durable --cluster-durable --file-size=128 --file-count=64
Queue36 --durable --cluster-durable --file-size=128 --file-count=64
Queue37 --durable --file-size=24 --file-count=8
Queue38 --durable --cluster-durable --file-size=128 --file-count=64
Queue39 --durable --file-size=24 --file-count=8
Queue40 --durable --file-size=24 --file-count=8
Queue41 --durable --cluster-durable --file-size=24 --file-count=8
Queue42 --durable --file-size=128 --file-count=64
Queue43 --durable --cluster-durable --file-size=128 --file-count=64
Queue44 --durable --cluster-durable --file-size=128 --file-count=64
Queue45 --durable --cluster-durable --file-size=128 --file-count=64
So we have 20 queues with --file-count=64 option and 25 queues with --file-count=8 option.
According to the formula nfh=nq * (nf + 1) to estimate the number of file handles from the following link http://www.redhat.com/f/pdf/Large_Journal.pdf
nfh=20 * (64 + 1) + 25 * (8 + 1) = 1525
But the maximum number of the open files for qpid is set to 16384 that is ten times greater.
Best Regards,
Sergey Zhemzhitsky
Information Technology Division
Troika Dialog, 4, Romanov lane, Moscow 125009, Russia
Phone. +7 495 2580500 ext. 1246
-----Original Message-----
From: Gordon Sim [mailto:gsim@redhat.com]
Sent: Tuesday, August 02, 2011 1:42 PM
To: users@qpid.apache.org
Subject: Re: Too many open files
On 08/01/2011 11:53 AM, Zhemzhitsky Sergey wrote:
> We have already set ulimit to be able to open 16384 files, but the error still happens from time to time.
> Could you help us to identify the possible sources of this issue.
How many connections and durable queues do you expect to have? Each durable queue uses 4 file handles at least (depending on configuration, and assuming you are using the qpidcomponents.org store plugin). Each connection will also use a file handle.
Is it possible for example that you have e.g. stale queues that are no longer needed but taking up file handles?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Too many open files
Posted by Gordon Sim <gs...@redhat.com>.
On 08/01/2011 11:53 AM, Zhemzhitsky Sergey wrote:
> We have already set ulimit to be able to open 16384 files, but the error still happens from time to time.
> Could you help us to identify the possible sources of this issue.
How many connections and durable queues do you expect to have? Each
durable queue uses 4 file handles at least (depending on configuration,
and assuming you are using the qpidcomponents.org store plugin). Each
connection will also use a file handle.
Is it possible for example that you have e.g. stale queues that are no
longer needed but taking up file handles?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: Too many open files
Posted by Zhemzhitsky Sergey <Se...@troika.ru>.
Hi Pavel,
Here is an output from ulimit -a for qpidd user
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 73728
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 16384
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 73728
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
As you can see the maximum number of open files is 16384.
lsof -p $(pgrep qpidd) | nl gives about 4200 lines
Unfortunately it is difficult to understand why such an error occured because there is no output of lsof at the time it happened.
Best Regards,
Sergey Zhemzhitsky
Information Technology Division
Troika Dialog, 4, Romanov lane, Moscow 125009, Russia
Phone. +7 495 2580500 ext. 1246
-----Original Message-----
From: Pavel Moravec [mailto:pmoravec@redhat.com]
Sent: Tuesday, August 02, 2011 12:33 PM
To: users@qpid.apache.org
Subject: Re: Too many open files
Hi Sergey,
no, this error log comes from OS, not from qpidd process. The system does not allow to open a new file descriptor as the qpidd process has opened maximum allowed FDs already.
What is your ulimit settings (ulimit -a for qpidd user)? What returns the following lsof command?
lsof -p $(pgrep qpidd) | nl
Kind regards,
Pavel
----- Original Message -----
From: "Zhemzhitsky Sergey" <Se...@troika.ru>
To: users@qpid.apache.org
Sent: Tuesday, August 2, 2011 9:39:12 AM
Subject: RE: Too many open files
Hi Pavel,
I have the following errors in the log file
error Could not accept socket: Too many open files (qpid/sys/posix/Socket.cpp:220) error Could not accept socket: Too many open files (qpid/sys/posix/Socket.cpp:220) error Could not accept socket: Too many open files (qpid/sys/posix/Socket.cpp:220)
Could it be connected with max connection count?
Best Regards,
Sergey Zhemzhitsky
Information Technology Division
Troika Dialog, 4, Romanov lane, Moscow 125009, Russia Phone. +7 495 2580500 ext. 1246
-----Original Message-----
From: Pavel Moravec [mailto:pmoravec@redhat.com]
Sent: Monday, August 01, 2011 3:42 PM
To: users@qpid.apache.org
Subject: Re: Too many open files
Hi Sergey,
what precise log did appear? Something like:
"Client max connection count limit exceeded: 500 connection refused" ?
If so, let see option
--max-connections N (500) Sets the maximum allowed
connections
Kind regards,
Pavel
----- Original Message -----
From: "Zhemzhitsky Sergey" <Se...@troika.ru>
To: users@qpid.apache.org
Sent: Monday, August 1, 2011 12:53:09 PM
Subject: Too many open files
Hi there,
We have already set ulimit to be able to open 16384 files, but the error still happens from time to time.
Could you help us to identify the possible sources of this issue.
Best Regards,
Sergey Zhemzhitsky
Information Technology Division
Troika Dialog, 4, Romanov lane, Moscow 125009, Russia Phone. +7 495 2580500 ext. 1246
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
Re: Too many open files
Posted by Pavel Moravec <pm...@redhat.com>.
Hi Sergey,
no, this error log comes from OS, not from qpidd process. The system does not allow to open a new file descriptor as the qpidd process has opened maximum allowed FDs already.
What is your ulimit settings (ulimit -a for qpidd user)? What returns the following lsof command?
lsof -p $(pgrep qpidd) | nl
Kind regards,
Pavel
----- Original Message -----
From: "Zhemzhitsky Sergey" <Se...@troika.ru>
To: users@qpid.apache.org
Sent: Tuesday, August 2, 2011 9:39:12 AM
Subject: RE: Too many open files
Hi Pavel,
I have the following errors in the log file
error Could not accept socket: Too many open files (qpid/sys/posix/Socket.cpp:220)
error Could not accept socket: Too many open files (qpid/sys/posix/Socket.cpp:220)
error Could not accept socket: Too many open files (qpid/sys/posix/Socket.cpp:220)
Could it be connected with max connection count?
Best Regards,
Sergey Zhemzhitsky
Information Technology Division
Troika Dialog, 4, Romanov lane, Moscow 125009, Russia
Phone. +7 495 2580500 ext. 1246
-----Original Message-----
From: Pavel Moravec [mailto:pmoravec@redhat.com]
Sent: Monday, August 01, 2011 3:42 PM
To: users@qpid.apache.org
Subject: Re: Too many open files
Hi Sergey,
what precise log did appear? Something like:
"Client max connection count limit exceeded: 500 connection refused" ?
If so, let see option
--max-connections N (500) Sets the maximum allowed
connections
Kind regards,
Pavel
----- Original Message -----
From: "Zhemzhitsky Sergey" <Se...@troika.ru>
To: users@qpid.apache.org
Sent: Monday, August 1, 2011 12:53:09 PM
Subject: Too many open files
Hi there,
We have already set ulimit to be able to open 16384 files, but the error still happens from time to time.
Could you help us to identify the possible sources of this issue.
Best Regards,
Sergey Zhemzhitsky
Information Technology Division
Troika Dialog, 4, Romanov lane, Moscow 125009, Russia Phone. +7 495 2580500 ext. 1246
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: Too many open files
Posted by Zhemzhitsky Sergey <Se...@troika.ru>.
Hi Pavel,
I have the following errors in the log file
error Could not accept socket: Too many open files (qpid/sys/posix/Socket.cpp:220)
error Could not accept socket: Too many open files (qpid/sys/posix/Socket.cpp:220)
error Could not accept socket: Too many open files (qpid/sys/posix/Socket.cpp:220)
Could it be connected with max connection count?
Best Regards,
Sergey Zhemzhitsky
Information Technology Division
Troika Dialog, 4, Romanov lane, Moscow 125009, Russia
Phone. +7 495 2580500 ext. 1246
-----Original Message-----
From: Pavel Moravec [mailto:pmoravec@redhat.com]
Sent: Monday, August 01, 2011 3:42 PM
To: users@qpid.apache.org
Subject: Re: Too many open files
Hi Sergey,
what precise log did appear? Something like:
"Client max connection count limit exceeded: 500 connection refused" ?
If so, let see option
--max-connections N (500) Sets the maximum allowed
connections
Kind regards,
Pavel
----- Original Message -----
From: "Zhemzhitsky Sergey" <Se...@troika.ru>
To: users@qpid.apache.org
Sent: Monday, August 1, 2011 12:53:09 PM
Subject: Too many open files
Hi there,
We have already set ulimit to be able to open 16384 files, but the error still happens from time to time.
Could you help us to identify the possible sources of this issue.
Best Regards,
Sergey Zhemzhitsky
Information Technology Division
Troika Dialog, 4, Romanov lane, Moscow 125009, Russia Phone. +7 495 2580500 ext. 1246
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
Re: Too many open files
Posted by Pavel Moravec <pm...@redhat.com>.
Hi Sergey,
what precise log did appear? Something like:
"Client max connection count limit exceeded: 500 connection refused" ?
If so, let see option
--max-connections N (500) Sets the maximum allowed
connections
Kind regards,
Pavel
----- Original Message -----
From: "Zhemzhitsky Sergey" <Se...@troika.ru>
To: users@qpid.apache.org
Sent: Monday, August 1, 2011 12:53:09 PM
Subject: Too many open files
Hi there,
We have already set ulimit to be able to open 16384 files, but the error still happens from time to time.
Could you help us to identify the possible sources of this issue.
Best Regards,
Sergey Zhemzhitsky
Information Technology Division
Troika Dialog, 4, Romanov lane, Moscow 125009, Russia
Phone. +7 495 2580500 ext. 1246
_______________________________________________________
The information contained in this message may be privileged and conf idential and protected from disclosure. If you are not the original intended recipient, you are hereby notified that any review, retransmission, dissemination, or other use of, or taking of any action in reliance upon, this information is prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and delete it from your computer. Thank you for your cooperation. Troika Dialog, Russia.
If you need assistance please contact our Contact Center (+7495) 258 0500 or go to www.troika.ru/eng/Contacts/system.wbp
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org