You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by David O'Connor D <da...@ericsson.com> on 2013/09/22 12:37:14 UTC
ActiveMQ 5.5.1 - Connections stuck in CLOSE_WAIT state, reaching
file descriptor limit
Hi,
There has been an issue seen where ActiveMQ 5.5.1 would keep outbound connections in a CLOSE_WAIT state, to the point where the application reaches the file descriptor limit.
The broker uses the Openwire format on TCP with closeAsync set to false. It is also non-persistant, storing all messages within memory.
The fault is infrequent, I've been unable to reproduce this on-demand. I've searched and I've been unable to see any re-occurance of this issue on the same version.
I've read that this issue was seen before in previous releases, could this be the same issue seen?
Regards,
David
Re: ActiveMQ 5.5.1 - Connections stuck in CLOSE_WAIT state,
reaching file descriptor limit
Posted by David O'C <mr...@gmail.com>.
As mentioned before, the broker wasn't at the maximum heap limit. The
heapdump was manually created using jmap but by using the -F option as
ActiveMQ wouldn't accept even JMX connections.
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-5-1-Connections-stuck-in-CLOSE-WAIT-state-reaching-file-descriptor-limit-tp4671744p4676700.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ 5.5.1 - Connections stuck in CLOSE_WAIT state,
reaching file descriptor limit
Posted by artnaseef <ar...@artnaseef.com>.
For the broker memory - is that looking at the Heap Size and Permgen Size,
checking for broker producer-flow-contorl-messages, or something else?
Not being able to connect JMX is a typical symptom of a JVM that's out of
Heap or Permgen space. If you connect jconsole, visualvm, or the like
before the problem occurs, it will be possible to see the memory status at
the time of the problem.
The following JVM option can help with detecting out-of-memory conditions:
-XX:+HeapDumpOnOutOfMemoryError
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-5-1-Connections-stuck-in-CLOSE-WAIT-state-reaching-file-descriptor-limit-tp4671744p4676653.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ 5.5.1 - Connections stuck in CLOSE_WAIT state,
reaching file descriptor limit
Posted by David O'C <mr...@gmail.com>.
The broker wasn't at its memory limit at the time when this had occurred, but
it wouldn't allow for JMX connections. I'd assume
I was able to collect a heapdump before ActiveMQ was restarted and it does
mention the Server Transport thread mentioned earlier. I'm not able to
identify the thread to the stacks in the thread dump:
<http://activemq.2283324.n4.nabble.com/file/n4676637/2014-01-22_14_37_06-Eclipse_Memory_Analyzer.jpg>
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-5-1-Connections-stuck-in-CLOSE-WAIT-state-reaching-file-descriptor-limit-tp4671744p4676637.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ 5.5.1 - Connections stuck in CLOSE_WAIT state, reaching
file descriptor limit
Posted by Gary Tully <ga...@gmail.com>.
I have seen something like this when the hand off thread that deals
with accepted sockets die. The error handling needs to be improved to
make that case more obvious.
It is a long shot, but you could disable the use of an accept queue
and see if that resolves the problem.
<transportConnector name="openwire"
uri="<host>:<port>?useQueueForAccept=false"/>
The thread name (from trunk) that handles accepted sockets server side
is prefixed with "ActiveMQ Transport Server Thread Handler:" so if
that is not visible, which seems to be the case in a cursory look at
the jstack, you could be in business.
useQueueForAccept=true allows the broker to handle a connection storm
without a tcp timeout, however clients typically retry so it is not a
big deal if it is set to false.
On 21 January 2014 19:50, David O'C <mr...@gmail.com> wrote:
> Hi Christian,
>
> Apologies for the late reply. I've a thread dump of the broker when it had
> reached the limit for open sockets.
>
> actmqjstack.txt
> <http://activemq.2283324.n4.nabble.com/file/n4676596/actmqjstack.txt>
>
> This had to be taken with force with jstack as ActiveMQ at that stage would
> not accept any JMX connections.
>
> Regards,
> David
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-5-1-Connections-stuck-in-CLOSE-WAIT-state-reaching-file-descriptor-limit-tp4671744p4676596.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
http://redhat.com
http://blog.garytully.com
Re: ActiveMQ 5.5.1 - Connections stuck in CLOSE_WAIT state,
reaching file descriptor limit
Posted by artnaseef <ar...@artnaseef.com>.
Without looking at the stack trace carefully, problems getting stack traces
is often seen with a memory leak or overloaded broker (i.e. running out of
heap or permgen space).
Can you connect jconsole or visualvm to the broker and check the heap and
permgen space when the problem occurs? If you want to capture JVM stats to
a text file, jstat is good for that.
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-5-1-Connections-stuck-in-CLOSE-WAIT-state-reaching-file-descriptor-limit-tp4671744p4676604.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ 5.5.1 - Connections stuck in CLOSE_WAIT state,
reaching file descriptor limit
Posted by David O'C <mr...@gmail.com>.
Hi Christian,
Apologies for the late reply. I've a thread dump of the broker when it had
reached the limit for open sockets.
actmqjstack.txt
<http://activemq.2283324.n4.nabble.com/file/n4676596/actmqjstack.txt>
This had to be taken with force with jstack as ActiveMQ at that stage would
not accept any JMX connections.
Regards,
David
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-5-1-Connections-stuck-in-CLOSE-WAIT-state-reaching-file-descriptor-limit-tp4671744p4676596.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ 5.5.1 - Connections stuck in CLOSE_WAIT state, reaching
file descriptor limit
Posted by Christian Posta <ch...@gmail.com>.
When you see this next, can you take stack traces and post back here?
On Sun, Sep 22, 2013 at 3:37 AM, David O'Connor D <
david.d.o.connor@ericsson.com> wrote:
> Hi,
>
> There has been an issue seen where ActiveMQ 5.5.1 would keep outbound
> connections in a CLOSE_WAIT state, to the point where the application
> reaches the file descriptor limit.
>
> The broker uses the Openwire format on TCP with closeAsync set to false.
> It is also non-persistant, storing all messages within memory.
>
> The fault is infrequent, I've been unable to reproduce this on-demand.
> I've searched and I've been unable to see any re-occurance of this issue
> on the same version.
>
> I've read that this issue was seen before in previous releases, could this
> be the same issue seen?
>
> Regards,
> David
>
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta