ActiveMQ 5.6 CPU steadily increases but drops at noon and midnight exactly?

[secondspan <mr...@gmail.com>]

Hi all,

We've been having a strange issue where the CPU load on our ActiveMQ server
increases continuously, only to suddenly drop off like clockwork at noon and
midnight sharp.  We are using JDK 1.7_04 with the G1 collector.

The server is receiving barely any traffic from 8pm-5am, so the CPU load
appears completely unrelated to the rate at which we process messages
through ActiveMQ.   Also, 10am-4pm is when it is processing the most
messages, so it's odd that it's at the lowest CPU usage around 12:10pm in
the middle of this peak time.

At first I suspected run-away NIO threads that are furiously polling, but it
does appear that ActiveMQ is cleaning up and constantly re-allocating new
"NIO Worker" threads and in examining their stacktrace they are all happily
spending most of their time here:

sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)


Here you can see the "Top Threads" using CPU as reported by the top threads
plug-in for JConsole.  The top threads are always mostly the "NIO Worker"
threads.

http://activemq.2283324.n4.nabble.com/file/n4654328/alertmq01_nio_cpu_12hrs.jpg 

Here you can see like clockwork the CPU drops at noon and midnight.  The
23rd is the exception, where the drop-offs that didn't happen at noon and
midnight were manual restarts of the server trying various additional
settings.

http://activemq.2283324.n4.nabble.com/file/n4654328/cpu-week.png 


For kicks, I tried the 5.7-SNAPSHOT from July 23rd.  It didn't affect the
gradual increase in CPU load.

We are using "-Dorg.apache.activemq.UseDedicatedTaskRunner=false"

Here's the line from activemq.xml where we define the transportConnector:

<transportConnector name="openwire"
uri="nio://0.0.0.0:61616?transport.closeAsync=false&amp;soTimeout=60000&amp;wireFormat.maxInactivityDurationInitalDelay=60000&amp;keepAlive=true"/>


I'm likely to try reverting back to BIO next as I have a hunch this might be
similar to the NIO problems that Jetty used to have back on early JDK 1.6
flavors.

I figure to try the forums to see if anyone had any thoughts...

Cheers,
Mike




--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-6-CPU-steadily-increases-but-drops-at-noon-and-midnight-exactly-tp4654328.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ 5.6 CPU steadily increases but drops at noon and midnight exactly?

[secondspan <mr...@gmail.com>]

Downgrading to JDK 1.6 from 1.7 resolves the strange ever-growing CPU
problem.

Interestingly, the top thread in terms of CPU usage becomes the "Selector
Worker: 0" after the downgrade.

http://activemq.2283324.n4.nabble.com/file/n4654368/mq_topThreads_16.png 

I agree it feels like a livelock of some sort that spontaneously resolves
itself every 12 hours.  

I can foresee spending some more time doing load testing against an ActiveMQ
instance on JDK 1.7 in a staging environment in the future with extra
logging enabled, but we're likely to defer any further investigation since
running an older JDK allows us to move onto other production matters and
defer this problem until the broader community attempts to run ActiveMQ on
1.7.



--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-6-CPU-steadily-increases-but-drops-at-noon-and-midnight-exactly-tp4654328p4654368.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ 5.6 CPU steadily increases but drops at noon and midnight exactly?

[secondspan <mr...@gmail.com>]

We've downgraded the JVM to a 1.6 from 1.7 and are observing the CPU usage
today.

We run many, many instances of ActiveMQ and this lone server that was using
JVM 1.7 is the one with the issue.  The other variable is the machine with
the issue is using a 64-bit JVM vs a 32-bit JVM.

We run many different server-side Java apps using 1.7 and for the most part
they actually all behave far better under 1.7 (the most obvious one being
Tomcat 7), so there's something between the combination of ActiveMQ and our
configuration/environment on that box that causes the CPU problem.



--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-6-CPU-steadily-increases-but-drops-at-noon-and-midnight-exactly-tp4654328p4654363.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ 5.6 CPU steadily increases but drops at noon and midnight exactly?

[Gaurav Sharma <ga...@gmail.com>]

This seems like an interesting problem and assuming it is no longer ntp
related, it seems to hint at a livelock but cannot say anything
definitively without setup information. Can you pastebin or gist  jstack -F
<pid> output and also your config's? This of interest to me and others
planning to switch to Java 7 after the impending EOL later this year. You
might also want to turn up activemq's rootLogger to DEBUG to get more clues.

On Tue, Jul 24, 2012 at 7:49 PM, secondspan <mr...@gmail.com> wrote:

>
> Switched from NIO to BIO.
>
> The problem still continues...
>
> CPU load increases over time despite the fact that traffic per minute going
> into the server is going down.
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-5-6-CPU-steadily-increases-but-drops-at-noon-and-midnight-exactly-tp4654328p4654331.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: ActiveMQ 5.6 CPU steadily increases but drops at noon and midnight exactly?

[secondspan <mr...@gmail.com>]

Switched from NIO to BIO.

The problem still continues... 

CPU load increases over time despite the fact that traffic per minute going
into the server is going down.




--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-5-6-CPU-steadily-increases-but-drops-at-noon-and-midnight-exactly-tp4654328p4654331.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.