You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by JacobS <ja...@gmail.com> on 2011/11/07 16:31:26 UTC
clients are getting disconnected
I am using version 5.5.0 and after connecting 1000 clients with nio transport
connector, clients start to disconnect with the exception:
15:24:11,307 WARN FailoverTransport:231 - Transport (null) failed to
nio://server2:61618 , attempting to automatically reconnect due to:
org.apache.activemq.transport.InactivityIOException: Channel was inactive
for too (>30000) long: server2/192.168.5.192:61618 [{InactivityMonitor Async
Task: java.util.concurrent.ThreadPoolExecutor$Worker@2083fc73}]
And in the server I am getting these INFO messages:
17:14:46.808 [ActiveMQ NIO Worker] INFO o.a.a.broker.TransportConnection -
The connection to 'null' is taking a long time to shutdown.
after querying a heap dump using Eclipse Memory Analyzer.
Leak Suspects reports:
1,893 instances of
"org.apache.activemq.broker.region.cursors.FilePendingMessageCursor", loaded
by "org.apache.activemq.activemq-core" occupy 502,260,984 (68.64%) bytes.
These instances are referenced from one instance of "java.lang.Object[]",
loaded by "<system class loader>"
Keywords
java.lang.Object[]
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor
org.apache.activemq.activemq-core
I seems that although I am using nio each connection is consuming a lot of
memory.
How can I reduce memory consumption ?
--
View this message in context: http://activemq.2283324.n4.nabble.com/clients-are-getting-disconnected-tp3999308p3999308.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: clients are getting disconnected
Posted by JacobS <ja...@gmail.com>.
Is the 'one FilePendingMessageCursor (minimum 256K) x Topic Consumer', memory
usage by design ?
It seems that its a lot of wasted memory, Hundreds Of Megabytes are wasted
on those 2 arrays in OpenWireFormat object ( which are all null at the point
when the client creates the topic consumers any way ).
I am looking for a way I can connect thousands of clients to a single broker
without having to require to much RAM on the brokers machine.
Can tweaking 'memoryLimit' reduce the memory used by the broker ?
--
View this message in context: http://activemq.2283324.n4.nabble.com/clients-are-getting-disconnected-tp3999308p4019592.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: clients are getting disconnected
Posted by Dejan Bosanac <de...@nighttale.net>.
Try tweaking memoryLimit option on destination policy and/or systemUsage
Regards
--
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Tue, Nov 8, 2011 at 6:08 PM, JacobS <ja...@gmail.com> wrote:
> After some more heap dump investigation I found what is eating all the
> memory.
> The dump shows that I have more than 2000 TopicSubscription objects and for
> each one there is an FilePendingMessageCursor, which consumes 265K (the
> OpenWireFormat object has 2 arrays each consuming 130K).
> My question is, is there any way to reduce the amount of
> FilePendingMessageCursor objects ?
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/clients-are-getting-disconnected-tp3999308p4016754.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: clients are getting disconnected
Posted by JacobS <ja...@gmail.com>.
After some more heap dump investigation I found what is eating all the
memory.
The dump shows that I have more than 2000 TopicSubscription objects and for
each one there is an FilePendingMessageCursor, which consumes 265K (the
OpenWireFormat object has 2 arrays each consuming 130K).
My question is, is there any way to reduce the amount of
FilePendingMessageCursor objects ?
--
View this message in context: http://activemq.2283324.n4.nabble.com/clients-are-getting-disconnected-tp3999308p4016754.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: clients are getting disconnected
Posted by JacobS <ja...@gmail.com>.
Can this be related to :
ActiveMQ.Advisory.TempQueue,ActiveMQ.Advisory.TempTopic destination ?
I added a breakpoint to the creation of FilePendingMessageCursor in the
TopicSubscription class
and I can see that FilePendingMessageCursor is created 5 times when a client
connects ( and creates consumers ).
2 of the instances are created for topics that the client is consuming.
but 3 more instances are created with ConsumerInfo.destination value of
"ActiveMQ.Advisory.TempQueue,ActiveMQ.Advisory.TempTopic".
why are these advisories needed. the client is not consuming any messages
from these destinations ?
--
View this message in context: http://activemq.2283324.n4.nabble.com/clients-are-getting-disconnected-tp3999308p4015903.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: clients are getting disconnected
Posted by JacobS <ja...@gmail.com>.
I don't use durable topic subscribers at all
--
View this message in context: http://activemq.2283324.n4.nabble.com/clients-are-getting-disconnected-tp3999308p4015777.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: clients are getting disconnected
Posted by Dejan Bosanac <de...@nighttale.net>.
They are used for storing temporary messages in destinations and durable
subscribers, see
http://activemq.apache.org/message-cursors.html
Do you have a large number of durable topic subscribers?
Regards
--
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Tue, Nov 8, 2011 at 12:10 PM, JacobS <ja...@gmail.com> wrote:
> I don't have many destinations.
> what is org.apache.activemq.broker.region.cursors.FilePendingMessageCursor
> ?
> it is consuming most of the memory.
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/clients-are-getting-disconnected-tp3999308p4015627.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: clients are getting disconnected
Posted by JacobS <ja...@gmail.com>.
I don't have many destinations.
what is org.apache.activemq.broker.region.cursors.FilePendingMessageCursor ?
it is consuming most of the memory.
--
View this message in context: http://activemq.2283324.n4.nabble.com/clients-are-getting-disconnected-tp3999308p4015627.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: clients are getting disconnected
Posted by Dejan Bosanac <de...@nighttale.net>.
Can you create a test case that reproduces it? Also, are you using a large
number of destinations (in addition to large number of connections)? If,
you might find this useful
http://activemq.apache.org/how-do-i-configure-10s-of-1000s-of-queues-in-a-single-broker-.html
Regards
--
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Mon, Nov 7, 2011 at 4:31 PM, JacobS <ja...@gmail.com> wrote:
> I am using version 5.5.0 and after connecting 1000 clients with nio
> transport
> connector, clients start to disconnect with the exception:
>
> 15:24:11,307 WARN FailoverTransport:231 - Transport (null) failed to
> nio://server2:61618 , attempting to automatically reconnect due to:
> org.apache.activemq.transport.InactivityIOException: Channel was inactive
> for too (>30000) long: server2/192.168.5.192:61618 [{InactivityMonitor
> Async
> Task: java.util.concurrent.ThreadPoolExecutor$Worker@2083fc73}]
>
> And in the server I am getting these INFO messages:
>
> 17:14:46.808 [ActiveMQ NIO Worker] INFO o.a.a.broker.TransportConnection -
> The connection to 'null' is taking a long time to shutdown.
>
> after querying a heap dump using Eclipse Memory Analyzer.
> Leak Suspects reports:
>
> 1,893 instances of
> "org.apache.activemq.broker.region.cursors.FilePendingMessageCursor",
> loaded
> by "org.apache.activemq.activemq-core" occupy 502,260,984 (68.64%) bytes.
> These instances are referenced from one instance of "java.lang.Object[]",
> loaded by "<system class loader>"
>
> Keywords
> java.lang.Object[]
> org.apache.activemq.broker.region.cursors.FilePendingMessageCursor
> org.apache.activemq.activemq-core
>
>
> I seems that although I am using nio each connection is consuming a lot of
> memory.
>
> How can I reduce memory consumption ?
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/clients-are-getting-disconnected-tp3999308p3999308.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>