You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "Tlholoe, Peter" <PT...@fnb.co.za> on 2012/08/27 08:14:13 UTC
Sounds like ActiveMQ still hangs after memory increase
Good Day,
We have increased the maximum heap size of the JVM started by ActiveMQ to 2048mb, and set the systemUsage memoryLimit to 1280 (70% of the total max heap size). We have also set the per destination policy. For topics (">") since they are heavily used we have set the memoryLimit to 512 and 128 mb for Queues (">"). And we have also disabled producer flow control. Can it be something else ?. We are using ActiveMQ 5.4.2
I have attached our activemq.xml
To read FirstRand Bank's Disclaimer for this email click on the following address or copy into your Internet browser:
https://www.fnb.co.za/disclaimer.html
If you are unable to access the Disclaimer, send a blank e-mail to
firstrandbankdisclaimer@fnb.co.za and we will send you a copy of the Disclaimer.
RE: Sounds like ActiveMQ still hangs after memory increase
Posted by "Tlholoe, Peter" <PT...@fnb.co.za>.
Hi Sharma
I will try that (turn off the dedicated task runner) and advise, please see below the answers to the questions that you have asked.
1. how many processors/cores do your machines have?
16 cores
2. are your clients (producers/consumers) local to the broker on the same machine but different process?
All our consumers are remote, different applications sitting in different departments (but in the same DMZ). Some of our producers are local and some remote. We use these messages to kick
Different business work flows in different fulfilment areas
3. looking at the oracle-ds settings in config, is the same machine also hosting oracle?
The Oracle database is sitting in a different machine but in the same DMZ
4. what nic does the machine have?
I will have to find this out from our infrastructure guys.
Regards,
P
On Mon, Aug 27, 2012 at 12:38 AM, Tlholoe, Peter <PT...@fnb.co.za> wrote:
> Hi Gaurav,
> I did take thread dumps during hang, and after the fact. I have
> attached
> 10 thread dump file. T1dump -t7dump are during the hang and t7dump to
> t10dump are after the hang period.
>
> -----Original Message-----
> From: Gaurav Sharma [mailto:gaurav.cs.sharma@gmail.com]
> Sent: 27 August 2012 08:33 AM
> To: users@activemq.apache.org
> Cc: users@activemq.apache.org
> Subject: Re: Sounds like ActiveMQ still hangs after memory increase
>
> When the process hangs, please take a thread dump (kill -3) and
> pastebin/gist and share so folks on here can help. Also, how did you
> determine that it hangs - no log activity, jmx response, etc?
>
> By the way, 1-2gigs of heap is not very large these days not that you
> should just up it but just fyi.
>
> On Aug 26, 2012, at 23:14, "Tlholoe, Peter" <PT...@fnb.co.za> wrote:
>
> > Good Day,
> > We have increased the maximum heap size of the JVM started by
> > ActiveMQ
> to 2048mb, and set the systemUsage memoryLimit to 1280 (70% of the
> total max heap size). We have also set the per destination policy. For
> topics
> (">") since they are heavily used we have set the memoryLimit to 512
> and
> 128 mb for Queues (">"). And we have also disabled producer flow control.
> Can it be something else ?. We are using ActiveMQ 5.4.2
> >
> > I have attached our activemq.xml
> > To read FirstRand Bank's Disclaimer for this email click on the
> following address or copy into your Internet browser:
> > https://www.fnb.co.za/disclaimer.html
> >
> > If you are unable to access the Disclaimer, send a blank e-mail to
> firstrandbankdisclaimer@fnb.co.za and we will send you a copy of the
> Disclaimer.
> > <activemq.xml>
>
> To read FirstRand Bank's Disclaimer for this email click on the
> following address or copy into your Internet browser:
> https://www.fnb.co.za/disclaimer.html
>
> If you are unable to access the Disclaimer, send a blank e-mail to
> firstrandbankdisclaimer@fnb.co.za and we will send you a copy of the
> Disclaimer.
>
To read FirstRand Bank's Disclaimer for this email click on the following address or copy into your Internet browser:
https://www.fnb.co.za/disclaimer.html
If you are unable to access the Disclaimer, send a blank e-mail to
firstrandbankdisclaimer@fnb.co.za and we will send you a copy of the Disclaimer.
Re: Sounds like ActiveMQ still hangs after memory increase
Posted by Gaurav Sharma <ga...@gmail.com>.
That's a lot of threads. You might wanna turn off the dedicated task runner
via either the activemq connection factory (in code) or via config (
http://activemq.apache.org/how-do-i-configure-10s-of-1000s-of-queues-in-a-single-broker-.html
).
A couple of questions from the thread dump analysis:
1. how many processors/cores do your machines have?
2. are your clients (producers/consumers) local to the broker on the same
machine but different process?
3. looking at the oracle-ds settings in config, is the same machine also
hosting oracle?
4. what nic does the machine have?
On Mon, Aug 27, 2012 at 12:38 AM, Tlholoe, Peter <PT...@fnb.co.za> wrote:
> Hi Gaurav,
> I did take thread dumps during hang, and after the fact. I have attached
> 10 thread dump file. T1dump -t7dump are during the hang and t7dump to
> t10dump are after the hang period.
>
> -----Original Message-----
> From: Gaurav Sharma [mailto:gaurav.cs.sharma@gmail.com]
> Sent: 27 August 2012 08:33 AM
> To: users@activemq.apache.org
> Cc: users@activemq.apache.org
> Subject: Re: Sounds like ActiveMQ still hangs after memory increase
>
> When the process hangs, please take a thread dump (kill -3) and
> pastebin/gist and share so folks on here can help. Also, how did you
> determine that it hangs - no log activity, jmx response, etc?
>
> By the way, 1-2gigs of heap is not very large these days not that you
> should just up it but just fyi.
>
> On Aug 26, 2012, at 23:14, "Tlholoe, Peter" <PT...@fnb.co.za> wrote:
>
> > Good Day,
> > We have increased the maximum heap size of the JVM started by ActiveMQ
> to 2048mb, and set the systemUsage memoryLimit to 1280 (70% of the total
> max heap size). We have also set the per destination policy. For topics
> (“>”) since they are heavily used we have set the memoryLimit to 512 and
> 128 mb for Queues (“>”). And we have also disabled producer flow control.
> Can it be something else ?. We are using ActiveMQ 5.4.2
> >
> > I have attached our activemq.xml
> > To read FirstRand Bank's Disclaimer for this email click on the
> following address or copy into your Internet browser:
> > https://www.fnb.co.za/disclaimer.html
> >
> > If you are unable to access the Disclaimer, send a blank e-mail to
> firstrandbankdisclaimer@fnb.co.za and we will send you a copy of the
> Disclaimer.
> > <activemq.xml>
>
> To read FirstRand Bank's Disclaimer for this email click on the following
> address or copy into your Internet browser:
> https://www.fnb.co.za/disclaimer.html
>
> If you are unable to access the Disclaimer, send a blank e-mail to
> firstrandbankdisclaimer@fnb.co.za and we will send you a copy of the
> Disclaimer.
>
RE: Sounds like ActiveMQ still hangs after memory increase
Posted by "Tlholoe, Peter" <PT...@fnb.co.za>.
Hi Gaurav,
I did take thread dumps during hang, and after the fact. I have attached 10 thread dump file. T1dump -t7dump are during the hang and t7dump to t10dump are after the hang period.
-----Original Message-----
From: Gaurav Sharma [mailto:gaurav.cs.sharma@gmail.com]
Sent: 27 August 2012 08:33 AM
To: users@activemq.apache.org
Cc: users@activemq.apache.org
Subject: Re: Sounds like ActiveMQ still hangs after memory increase
When the process hangs, please take a thread dump (kill -3) and pastebin/gist and share so folks on here can help. Also, how did you determine that it hangs - no log activity, jmx response, etc?
By the way, 1-2gigs of heap is not very large these days not that you should just up it but just fyi.
On Aug 26, 2012, at 23:14, "Tlholoe, Peter" <PT...@fnb.co.za> wrote:
> Good Day,
> We have increased the maximum heap size of the JVM started by ActiveMQ to 2048mb, and set the systemUsage memoryLimit to 1280 (70% of the total max heap size). We have also set the per destination policy. For topics (“>”) since they are heavily used we have set the memoryLimit to 512 and 128 mb for Queues (“>”). And we have also disabled producer flow control. Can it be something else ?. We are using ActiveMQ 5.4.2
>
> I have attached our activemq.xml
> To read FirstRand Bank's Disclaimer for this email click on the following address or copy into your Internet browser:
> https://www.fnb.co.za/disclaimer.html
>
> If you are unable to access the Disclaimer, send a blank e-mail to firstrandbankdisclaimer@fnb.co.za and we will send you a copy of the Disclaimer.
> <activemq.xml>
To read FirstRand Bank's Disclaimer for this email click on the following address or copy into your Internet browser:
https://www.fnb.co.za/disclaimer.html
If you are unable to access the Disclaimer, send a blank e-mail to
firstrandbankdisclaimer@fnb.co.za and we will send you a copy of the Disclaimer.
Re: Sounds like ActiveMQ still hangs after memory increase
Posted by Gaurav Sharma <ga...@gmail.com>.
When the process hangs, please take a thread dump (kill -3) and pastebin/gist and share so folks on here can help. Also, how did you determine that it hangs - no log activity, jmx response, etc?
By the way, 1-2gigs of heap is not very large these days not that you should just up it but just fyi.
On Aug 26, 2012, at 23:14, "Tlholoe, Peter" <PT...@fnb.co.za> wrote:
> Good Day,
> We have increased the maximum heap size of the JVM started by ActiveMQ to 2048mb, and set the systemUsage memoryLimit to 1280 (70% of the total max heap size). We have also set the per destination policy. For topics (“>”) since they are heavily used we have set the memoryLimit to 512 and 128 mb for Queues (“>”). And we have also disabled producer flow control. Can it be something else ?. We are using ActiveMQ 5.4.2
>
> I have attached our activemq.xml
> To read FirstRand Bank's Disclaimer for this email click on the following address or copy into your Internet browser:
> https://www.fnb.co.za/disclaimer.html
>
> If you are unable to access the Disclaimer, send a blank e-mail to firstrandbankdisclaimer@fnb.co.za and we will send you a copy of the Disclaimer.
> <activemq.xml>