ActiveMQ 5.1.0 and 5.2.0 runs out of file descriptors due to CLOSE_WAIT bug

[vjavarap <ve...@javarappa.org>]

We have been experiencing severe issues with ActiveMQ 5.1.0 running out of
file descriptors and netstat reveals thousands of tcp connections stuck in
CLOSE_WAIT within a few minutes or hours or days of starting up based on the
event volume.

Clearly, other activemq users are seeing the same issue:
https://issues.apache.org/activemq/browse/AMQ-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel

The ActiveMQ runs on a Linux (CentOS) server. We are using stomp as the
transport connector and production and consumption are via PHP clients.

Problem Definition
- When we add thousands of events onto the queue with no consumption, the
events are registered and things work well.
- When events are consumed later when no events are being added at the same
time, things work smoothly.
- However, when events are being added and consumed at the same time, the
CLOSE_WAITs shoot up into the thousands (or tens of thousands) based on
event activity.

We have tried upgrading to ActiveMQ 5.2.0 and having this in the config:

<transportConnector name="stomp"
uri="stomp://localhost:61613?transport.closeAsync=false"/>

Once the CLOSE_WAITs shoot up into the thousands, the event queue stops
accepting events.

I've set the ulimit for open files to 32K and we are nowhere close to the
upper limit.

Any thoughts?
-- 
View this message in context: http://www.nabble.com/ActiveMQ-5.1.0-and-5.2.0-runs-out-of-file-descriptors-due-to-CLOSE_WAIT-bug-tp22256050p22256050.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ 5.1.0 and 5.2.0 runs out of file descriptors due to CLOSE_WAIT bug

[Arjen van der Meijden <ac...@tweakers.net>]

Did you also remove the limits for queue sizes (most notably the one 
that limits all queues to only 5MB)?

With the ulimit up to 10240, the async closing turned off and the limits 
for the seperate queues removed (we just disabled the entire 
'destinationPolicy'-element from the default config) and the total 
storage size increased we haven't seen any of those problems.

Best regards,

Arjen

On 27-2-2009 23:54 vjavarap wrote:
> We have been experiencing severe issues with ActiveMQ 5.1.0 running out of
> file descriptors and netstat reveals thousands of tcp connections stuck in
> CLOSE_WAIT within a few minutes or hours or days of starting up based on the
> event volume.
> 
> Clearly, other activemq users are seeing the same issue:
> https://issues.apache.org/activemq/browse/AMQ-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel
> 
> The ActiveMQ runs on a Linux (CentOS) server. We are using stomp as the
> transport connector and production and consumption are via PHP clients.
> 
> Problem Definition
> - When we add thousands of events onto the queue with no consumption, the
> events are registered and things work well.
> - When events are consumed later when no events are being added at the same
> time, things work smoothly.
> - However, when events are being added and consumed at the same time, the
> CLOSE_WAITs shoot up into the thousands (or tens of thousands) based on
> event activity.
> 
> We have tried upgrading to ActiveMQ 5.2.0 and having this in the config:
> 
> <transportConnector name="stomp"
> uri="stomp://localhost:61613?transport.closeAsync=false"/>
> 
> Once the CLOSE_WAITs shoot up into the thousands, the event queue stops
> accepting events.
> 
> I've set the ulimit for open files to 32K and we are nowhere close to the
> upper limit.
> 
> Any thoughts?