AMQ 5.4.1 File Descriptor (FD) leak of ticots loopback networks

[mickhayes <mi...@gmail.com>]

Hi folks,

This (after plenty searching) is not a request for help so much as an early
start/nucleation point for this issue: somewhere to find and hang more info.
[Checked: We're not using Stomp + NIO; We do not have the symptoms of
AMQ-2999 (don't use peer transport.); There is no thread leak]

No reproducible test case yet, but when isolated, we will update. First seen
on migration to 5.4.1 from 4.1.1, but various changes in amq xml
configuration were made at the same time. We have some 4.1.1 clients
remaining, and understand that the combination of 4.1.1 client & 5.4.1
broker is "tested by the community" i.e. not tested by the Apache project.

Environment: Solaris 10 u9,  Java HotSpot(TM) 64-Bit Server VM version
17.0-b16.
Single, standalone broker 5.4.1 running as SMF service. Multiple clients on
same machine and others.
We have leaked up to 2000 ticots FDs so far on the busiest of our test
deployments - and it's only running approx. 5K messages per minute at the
moment. [During load test we would put of the order of 1M messages per
minute through the broker.]

Symptom: new FDs of the form (as seen from output of "lsof -p AMQpid | grep
ticots" ):
COMMAND  PID   USER   FD   TYPE        DEVICE    SIZE/OFF     NODE NAME
*java    *12345 some_user   someFDnum  *unix       **105*,xyz         *0t0*
*55050244* */devices/pseudo/tl@0:ticots->(socketpair:* 0xsomenum)
(0xsomenum)

I've bolded the non-varying parts.

This leak is suspected to be triggered by particular JMS/Openwire traffic.
We now have full DEBUG logging on and are trying to corellate the creation
of a new ticots FD with particular events in the log - and perhaps linking
the leak to a particular client.
Also using Dtrace to look at FD creation and disposal.

Changes from previous broker deployment, where the issue was not seen:
ActiveMQ version: 4.1.1 -> 5.4.1
TCP transport -> NIO transport (broker only change)
discardingDLQ plugin introduced.

All the best,
Mick Hayes.





-----
Michael Hayes B.Sc. (NUI), M.Sc. (DCU), SCSA SCNA 

--
View this message in context: http://activemq.2283324.n4.nabble.com/AMQ-5-4-1-File-Descriptor-FD-leak-of-ticots-loopback-networks-tp3653804p3653804.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: AMQ 5.4.1 File Descriptor (FD) leak of ticots loopback networks

[Dejan Bosanac <de...@nighttale.net>]

Hi,

is there a chance you can test nio with 5.5.0 or the latest 5.6-SNAPSHOT and
see if problem is still there? A test case would be great if you manage to
provide it.

Regards
-- 
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net


On Fri, Jul 29, 2011 at 9:18 AM, mickhayes <mi...@gmail.com> wrote:

> Update on this File Descriptor Leak with 5.4.1 broker and NIO transport -
> how
> to workaround/avoid.
> The leak *stops* when transport changed to TCP in broker xml.
> The leak *resumes *when transport changed back to NIO.
> I.e. this fault is only seen with NIO transport.
> I will update with a test case when I have one.
>
>
> -----
> Michael Hayes B.Sc. (NUI), M.Sc. (DCU), SCSA SCNA
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/AMQ-5-4-1-File-Descriptor-FD-leak-of-ticots-loopback-networks-tp3653804p3703254.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: AMQ 5.4.1 File Descriptor (FD) leak of ticots loopback networks

[mickhayes <mi...@gmail.com>]

Update on this File Descriptor Leak with 5.4.1 broker and NIO transport - how
to workaround/avoid.
The leak *stops* when transport changed to TCP in broker xml.
The leak *resumes *when transport changed back to NIO.
I.e. this fault is only seen with NIO transport.
I will update with a test case when I have one.


-----
Michael Hayes B.Sc. (NUI), M.Sc. (DCU), SCSA SCNA 

--
View this message in context: http://activemq.2283324.n4.nabble.com/AMQ-5-4-1-File-Descriptor-FD-leak-of-ticots-loopback-networks-tp3653804p3703254.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.