You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by peterchenadded <pe...@gmail.com> on 2014/04/13 08:19:05 UTC
Broker Memory Usage is 100% and tries to flush to disk with Null
Pointer Exception
Please help with the following issues we are encountering:
1. Broker Memory Usage is reaching 100%, we are not sure of the root cause,
however we believe it might be related to high load.
We have completed a heap dump and can see the percentUsage of one
org.apache.activemq.usage.MemoryUsage is 100 as seem below:
Type |Name |Value
-----------------------------------------------------------------------------------------
long |usage |268438348
ref |usageMutex |java.lang.Object @ 0x1160c2550
int |percentUsage |100
ref |parent |null
ref |limiter |org.apache.activemq.usage.DefaultUsageCapacity
@ 0x1160c2558
int |percentUsageMinDelta|1
ref |listeners |java.util.concurrent.CopyOnWriteArrayList @
0x1160c2568
boolean|debug |false
ref |name |default:memory
float |usagePortion |0.0
ref |children |java.util.concurrent.CopyOnWriteArrayList @
0x1160c25a0
ref |callbacks |java.util.LinkedList @ 0x1160c25d8
int |pollingTime |100
ref |started |java.util.concurrent.atomic.AtomicBoolean @
0x1160c2608
ref |executor |java.util.concurrent.ThreadPoolExecutor @
0x114eefd30
-----------------------------------------------------------------------------------------
We have configured the memoryUsage as 256mb and tried to put load in our
development environment by spamming the broker but the percentUsage did not
move from 0, but dequeue and enqueue counts went up quickly.
We would like to understand in what situations the memoryUsage is increased
and how to monitor for this without having to run jconsole or a flight
recording - jrmc.
2. Due to the issue above ActiveMQ tries to flush the messages to disk but
we get the following exception:
(org.apache.activemq.broker.region.cursors.FilePendingMessageCursor).444 -
Caught an IO Exception getting the DiskList
3135_PendingCursor:crossClusterTopicsOnly_ActiveMQ-broadcast-Name
java.lang.NullPointerException
at
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.getDiskList(FilePendingMessageCursor.java:442)
at
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.flushToDisk(FilePendingMessageCursor.java:423)
at
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.onUsageChanged(FilePendingMessageCursor.java:381)
at org.apache.activemq.usage.Usage$1.run(Usage.java:268)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Does a property value need to be set so that it will write to disk
successfully or since the messages are non-persistent a file cursor is in
compatible?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Memory-Usage-is-100-and-tries-to-flush-to-disk-with-Null-Pointer-Exception-tp4680254.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Memory Usage is 100% and tries to flush to disk with
Null Pointer Exception
Posted by peterchenadded <pe...@gmail.com>.
Please note we are using the following:
ActiveMQ: 5.5.1
Jrockit: Oracle JRockit(R) (build
R28.2.4-14-151097-1.6.0_33-20120618-1634-linux-x86_64, compiled mode)
artnaseef wrote
> If I understand correctly, the issue is reaching 100% memory usage. Are
> the attempts to increase memory usage by spamming the broker intended to
> reproduce the problem?
Yes that is correct, need to understand how the broker memory usage can be
increased sustained load. i.e. how is it counting the memory usage and are
there situations when the memory usage is count incorrectly?
artnaseef wrote
> To find where memory is being used, look at QueueSize, for Queues, and
> PendingQueueSize, for Topics, attributes of the Queue and Topic
> Subscription MBeans via JMX. Queue sizes can also be seen via the
> webconsole.
>
> Typically, 100% memory usage means slow consumption. With that in mind, I
> recommend collecting metrics with a JMX tool, such as jmxtrans, and
> feeding into a solution that can be used to monitor and report, such as
> one using OpenTSDB and Zabbix. Then monitor for slow consumption - either
> by setting limits to QueueSize and PendingQueueSize, or by monitoring
> AverageEnqueueTime.
We will have a look at these attributes and provide the numbers.
artnaseef wrote
> As for the NullPointerException - can you provide a small test case that
> reproduces the problem? Also, does it happen on the latest ActiveMQ
> builds?
One test case would be when the broker memory usage has exceeded 70% of it's
maximum and it needs to flush to disk. However, we are not able to produce
messages in such a way that the broker memory usage increases.
artnaseef wrote
> It's unclear why the attempts to reproduce (assuming that's what they are)
> do not see memory usage increase. One question - which memory usage value
> is being monitored? The broker's (top-level) memory usage, or memory
> usage of a specific destination?
We are measuring the memory usage of the broker in JConsole.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Memory-Usage-is-100-and-tries-to-flush-to-disk-with-Null-Pointer-Exception-tp4680254p4680264.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Broker Memory Usage is 100% and tries to flush to disk with
Null Pointer Exception
Posted by artnaseef <ar...@artnaseef.com>.
If I understand correctly, the issue is reaching 100% memory usage. Are the
attempts to increase memory usage by spamming the broker intended to
reproduce the problem?
To find where memory is being used, look at QueueSize, for Queues, and
PendingQueueSize, for Topics, attributes of the Queue and Topic Subscription
MBeans via JMX. Queue sizes can also be seen via the webconsole.
Typically, 100% memory usage means slow consumption. With that in mind, I
recommend collecting metrics with a JMX tool, such as jmxtrans, and feeding
into a solution that can be used to monitor and report, such as one using
OpenTSDB and Zabbix. Then monitor for slow consumption - either by setting
limits to QueueSize and PendingQueueSize, or by monitoring
AverageEnqueueTime.
As for the NullPointerException - can you provide a small test case that
reproduces the problem? Also, does it happen on the latest ActiveMQ builds?
It's unclear why the attempts to reproduce (assuming that's what they are)
do not see memory usage increase. One question - which memory usage value
is being monitored? The broker's (top-level) memory usage, or memory usage
of a specific destination?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Memory-Usage-is-100-and-tries-to-flush-to-disk-with-Null-Pointer-Exception-tp4680254p4680258.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.