You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Raagu <rk...@gmail.com> on 2014/06/05 14:56:55 UTC
Messages are getting struck in ActiveMQ v5.9.0 queue
Hi
I am facing problem of messages getting struck in activemq queue and
consumer is not able to consume messages when I pump around ~100 messages to
different queues.
I have configured ActiveMQ (v5.9.0) message broker in two separate linux
instances. For persistenceAdapter conf on both the instances, I have mounted
a NAS storage onto both the instances where activemq server is running.
Hence same NAS storage is mounted on both activemq instances at mount point
'/mnt/nas'. The size of the NAS storage is 20 GB.
So my persistenceAdapter conf looks like below
<persistenceAdapter>
<kahaDB directory="/mnt/nas" ignoreMissingJournalfiles="true"
checkForCorruptJournalFiles="true" checksumJournalFiles="true"/>
</persistenceAdapter>
The systemUsage configuration on both activemq server is like below
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap = "70"/>
</memoryUsage>
<storeUsage>
<storeUsage limit = "15 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit = "7 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
and I have enabled only 'tcp' transport connector
<transportConnector name = "openwire" uri =
"tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
When I see the partition details of the mount point '/mnt/nas' on both
activemq instances, I see following for the command df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda2 102953264 5840280 91883264 6% /
tmpfs 967652 0 967652 0% /dev/shm
/dev/xvda1 253871 52511 188253 22% /boot
//nas151.service.softlayer.com/IBM278684-16
139328579072 56369051136 82959527936 41% /mnt/nas
Hence I see 41% of /mnt/nas is used
The problem is when I start the activemq server (on both instances), I see
the following messages in the activemq.log
**************** START ***************
2014-06-05 12:48:40,350 | INFO |
PListStore:[/var/lib/apache-activemq-5.9.0/data/localhost/tmp_storage]
started | org.apache.activemq.store.kahadb.plist.PListStoreImpl | main
2014-06-05 12:48:40,454 | INFO | JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi |
org.apache.activemq.broker.jmx.ManagementContext | JMX connector
2014-06-05 12:48:40,457 | INFO | Using Persistence Adapter:
KahaDBPersistenceAdapter[/mnt/nas] |
org.apache.activemq.broker.BrokerService | main
2014-06-05 12:48:40,612 | INFO | Corrupt journal records found in
'/mnt/nas/db-1.log' between offsets: 8163..8209 |
org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,613 | INFO | Corrupt journal records found in
'/mnt/nas/db-1.log' between offsets: 12256..12327 |
org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,649 | INFO | Corrupt journal records found in
'/mnt/nas/db-1.log' between offsets: 20420..20585 |
org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,650 | INFO | Corrupt journal records found in
'/mnt/nas/db-1.log' between offsets: 28559..28749 |
org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,651 | INFO | Corrupt journal records found in
'/mnt/nas/db-1.log' between offsets: 32677..32842 |
org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,652 | INFO | Corrupt journal records found in
'/mnt/nas/db-1.log' between offsets: 36770..36960 |
org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,655 | INFO | Corrupt journal records found in
'/mnt/nas/db-1.log' between offsets: 49099..49264 |
org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,657 | INFO | Corrupt journal records found in
'/mnt/nas/db-1.log' between offsets: 61403..61474 |
org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,658 | INFO | Corrupt journal records found in
'/mnt/nas/db-1.log' between offsets: 65521..65567 |
org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,659 | INFO | Corrupt journal records found in
'/mnt/nas/db-1.log' between offsets: 69614..69685 |
org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,660 | INFO | Corrupt journal records found in
'/mnt/nas/db-1.log' between offsets: 77778..77824 |
org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:41,543 | INFO | KahaDB is version 5 |
org.apache.activemq.store.kahadb.MessageDatabase | main
2014-06-05 12:48:41,592 | INFO | Recovering from the journal ... |
org.apache.activemq.store.kahadb.MessageDatabase | main
2014-06-05 12:48:41,604 | INFO | Recovery replayed 66 operations from the
journal in 0.028 seconds. | org.apache.activemq.store.kahadb.MessageDatabase
| main
2014-06-05 12:48:41,772 | INFO | Apache ActiveMQ 5.9.0 (localhost,
ID:10.106.99.101-60576-1401972521638-0:1) is starting |
org.apache.activemq.broker.BrokerService | main
2014-06-05 12:48:41,892 | INFO | Listening for connections at:
tcp://10.106.99.101:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
| org.apache.activemq.transport.TransportServerThreadSupport | main
2014-06-05 12:48:41,893 | INFO | Connector openwire started |
org.apache.activemq.broker.TransportConnector | main
2014-06-05 12:48:41,893 | INFO | Apache ActiveMQ 5.9.0 (localhost,
ID:10.106.99.101-60576-1401972521638-0:1) started |
org.apache.activemq.broker.BrokerService | main
2014-06-05 12:48:41,893 | INFO | For help or more information please see:
http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
2014-06-05 12:48:41,897 | WARN | Store limit is 2048 mb, whilst the data
directory: /mnt/nas only has 0 mb of usable space - resetting to maximum
available disk space: 0 mb | org.apache.activemq.broker.BrokerService | main
2014-06-05 12:48:41,897 | ERROR | Store limit is 0 mb, whilst the max
journal file size for the store is: 32 mb, the store will not accept any
data when used. | org.apache.activemq.broker.BrokerService | main
******************** END **************
I see 'Corrupt journal records found in '/mnt/nas/db-1.log'. This comes for
everytime restart even though if I delete this file and restart.
I had put the flag to recover but still this log entry comes for every
restart.
Another problem is, even though my NAS storage is 20GB, it shows '/mnt/nas
only has 0 mb of usable space'. This is really weired. I dont know hw it can
see only 0 mb.
I request people here to give me some suggestions on why it is happening
like this and suggest me any better configurations to avoid messages getting
struck in queue.
Thanks
Raagu
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-are-getting-struck-in-ActiveMQ-v5-9-0-queue-tp4681717.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.