You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Prashant Katti <ka...@gmail.com> on 2017/07/20 06:25:34 UTC
Activemq Master dying without errors.
Hi All,
I have activemq 15.3.3 running on 2 hosts setup as Fault tolerant. Kahadb is
ZFS mount used by both active mq hosts. Seeing a weird issue, Master is
getting stopped without any errors in log file. I have attached both the
server logs. I am using default settings in activemq.xml
If you see below @2017-07-19 18:31:21 Master server was down, although
nothing is mentioned in the activemq.log file.
*Master Server :*
2017-07-19 18:31:21,235 | DEBUG | commit:
TX:ID:cmhvl014.limited.brands.com-38500-1500479355480-1:1:63891 syncCount: 2
| org.apache.activemq.transaction.LocalTransaction | ActiveMQ Transport:
tcp:///10.237.42.42:33178@61616
*2017-07-19 18:31:21,235 | DEBUG | commit:
TX:ID:cmhvl013.limited.brands.com-46039-1500479352933-1:1:64278 syncCount: 2
| org.apache.activemq.transaction.LocalTransaction | ActiveMQ Transport:
tcp:///10.237.42.41:40330@61616*
2017-07-19 18:53:56,883 | DEBUG | Using class path resource [activemq.xml]
from activemq.xml | org.apache.activemq.xbean.XBeanBrokerFactory | main
2017-07-19 18:53:56,926 | INFO | Refreshing
org.apache.activemq.xbean.XBeanBrokerFactory$1@7c16905e: startup date [Wed
Jul 19 18:53:56 EDT 2017]; root of context hierarchy |
org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
*Slave Server :*
2017-07-19 18:31:08,742 | DEBUG | Database
/message_broker/activemq/data/kahadb/lock is locked... waiting 10 seconds
for the database to be unlocked. Reason: java.io.IOException: File
'/message_broker/activemq/data/kahadb/lock' could not be locked. |
org.apache.activemq.store.SharedFileLocker | main
2017-07-19 18:31:18,817 | DEBUG | Database
/message_broker/activemq/data/kahadb/lock is locked... waiting 10 seconds
for the database to be unlocked. Reason: java.io.IOException: File
'/message_broker/activemq/data/kahadb/lock' could not be locked. |
org.apache.activemq.store.SharedFileLocker | main
*2017-07-19 18:31:48,853 | DEBUG | Store OpenWire version configured as: 11
| org.apache.activemq.store.kahadb.KahaDBStore | main*
2017-07-19 18:31:49,450 | DEBUG | Page File:
/message_broker/activemq/data/kahadb/db.data, Recovering page file... |
org.apache.activemq.store.kahadb.disk.page.PageFile | main
2017-07-19 18:31:49,491 | INFO | KahaDB is version 6 |
org.apache.activemq.store.kahadb.MessageDatabase | main
2017-07-19 18:31:49,492 | DEBUG | loading |
org.apache.activemq.store.kahadb.disk.index.BTreeIndex | main
Did anyone faced same issue?
Any help is highly appreciated.
Thanks,
Prashant
--
View this message in context: http://activemq.2283324.n4.nabble.com/Activemq-Master-dying-without-errors-tp4728716.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Activemq Master dying without errors.
Posted by Tim Bain <tb...@alumni.duke.edu>.
I can't say for sure that this is the cause of the behavior you're seeing,
but it's certainly something I'd recommend you fix. Keep in mind that the
heap is only part of the JVM's total RAM, and the JVM can only take part of
the host's RAM because the kernel needs some even if there aren't any other
user processes. On a 16GB host, if nothing else is running there, 12GB of
heap would be very safe, 14GB should work, and 15GB might work. But I'd
start with 12GB and see if your crashing behavior disappears, and you can
consider whether to increase the value later.
Tim
On Jul 20, 2017 1:09 AM, "Prashant Katti" <ka...@gmail.com> wrote:
> Thanks for the response Tim.
>
> The logging has been enabled with "debug" level. :(
>
> Still don't see that error.
>
> mq is running with 16GB of JVM
>
> *# Set jvm memory configuration (minimal/maximum amount of memory)
> ACTIVEMQ_OPTS_MEMORY="-Xms8G -Xmx16G"*
>
> [mule@lxp0172 kahadb]$ netstat -nap | grep 8161
> (Not all processes could be identified, non-owned process info
> will not be shown, you would have to be root to see it all.)
> tcp6 0 0 :::8161 :::* LISTEN
> 59849/java
> *[mule@lxp0172 kahadb]$ ps -ef | grep 59849
> mule 59849 1 4 Jul19 ? 00:10:28 /app/jdk1.8.0_91/bin/java
> -Xms8G -Xmx16G *
>
>
> and total memory on system is
>
> *[mule@lxp0172 bin]$ cat /proc/meminfo
> MemTotal: 16260324 kB*
> MemFree: 670892 kB
> MemAvailable: 7691660 kB
>
> Is this causing any issue?
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Activemq-Master-dying-without-errors-tp4728716p4728720.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Activemq Master dying without errors.
Posted by Prashant Katti <ka...@gmail.com>.
Thanks for the response Tim.
The logging has been enabled with "debug" level. :(
Still don't see that error.
mq is running with 16GB of JVM
*# Set jvm memory configuration (minimal/maximum amount of memory)
ACTIVEMQ_OPTS_MEMORY="-Xms8G -Xmx16G"*
[mule@lxp0172 kahadb]$ netstat -nap | grep 8161
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::8161 :::* LISTEN
59849/java
*[mule@lxp0172 kahadb]$ ps -ef | grep 59849
mule 59849 1 4 Jul19 ? 00:10:28 /app/jdk1.8.0_91/bin/java
-Xms8G -Xmx16G *
and total memory on system is
*[mule@lxp0172 bin]$ cat /proc/meminfo
MemTotal: 16260324 kB*
MemFree: 670892 kB
MemAvailable: 7691660 kB
Is this causing any issue?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Activemq-Master-dying-without-errors-tp4728716p4728720.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Activemq Master dying without errors.
Posted by Tim Bain <tb...@alumni.duke.edu>.
I've seen Java processes that run out of heap hard-crash without logging
anything (sometimes they do, sometimes not, even from run to run). Could
you simply have used more heap than your JVM had available?
Do you get any additional useful logging at DEBUG or TRACE?
Tim
On Thu, Jul 20, 2017 at 12:25 AM, Prashant Katti <ka...@gmail.com>
wrote:
> Hi All,
>
> I have activemq 15.3.3 running on 2 hosts setup as Fault tolerant. Kahadb
> is
> ZFS mount used by both active mq hosts. Seeing a weird issue, Master is
> getting stopped without any errors in log file. I have attached both the
> server logs. I am using default settings in activemq.xml
>
> If you see below @2017-07-19 18:31:21 Master server was down, although
> nothing is mentioned in the activemq.log file.
>
> *Master Server :*
>
> 2017-07-19 18:31:21,235 | DEBUG | commit:
> TX:ID:cmhvl014.limited.brands.com-38500-1500479355480-1:1:63891
> syncCount: 2
> | org.apache.activemq.transaction.LocalTransaction | ActiveMQ Transport:
> tcp:///10.237.42.42:33178@61616
> *2017-07-19 18:31:21,235 | DEBUG | commit:
> TX:ID:cmhvl013.limited.brands.com-46039-1500479352933-1:1:64278
> syncCount: 2
> | org.apache.activemq.transaction.LocalTransaction | ActiveMQ Transport:
> tcp:///10.237.42.41:40330@61616*
> 2017-07-19 18:53:56,883 | DEBUG | Using class path resource [activemq.xml]
> from activemq.xml | org.apache.activemq.xbean.XBeanBrokerFactory | main
> 2017-07-19 18:53:56,926 | INFO | Refreshing
> org.apache.activemq.xbean.XBeanBrokerFactory$1@7c16905e: startup date [Wed
> Jul 19 18:53:56 EDT 2017]; root of context hierarchy |
> org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
>
> *Slave Server :*
>
> 2017-07-19 18:31:08,742 | DEBUG | Database
> /message_broker/activemq/data/kahadb/lock is locked... waiting 10 seconds
> for the database to be unlocked. Reason: java.io.IOException: File
> '/message_broker/activemq/data/kahadb/lock' could not be locked. |
> org.apache.activemq.store.SharedFileLocker | main
> 2017-07-19 18:31:18,817 | DEBUG | Database
> /message_broker/activemq/data/kahadb/lock is locked... waiting 10 seconds
> for the database to be unlocked. Reason: java.io.IOException: File
> '/message_broker/activemq/data/kahadb/lock' could not be locked. |
> org.apache.activemq.store.SharedFileLocker | main
> *2017-07-19 18:31:48,853 | DEBUG | Store OpenWire version configured as: 11
> | org.apache.activemq.store.kahadb.KahaDBStore | main*
> 2017-07-19 18:31:49,450 | DEBUG | Page File:
> /message_broker/activemq/data/kahadb/db.data, Recovering page file... |
> org.apache.activemq.store.kahadb.disk.page.PageFile | main
> 2017-07-19 18:31:49,491 | INFO | KahaDB is version 6 |
> org.apache.activemq.store.kahadb.MessageDatabase | main
> 2017-07-19 18:31:49,492 | DEBUG | loading |
> org.apache.activemq.store.kahadb.disk.index.BTreeIndex | main
>
> Did anyone faced same issue?
>
> Any help is highly appreciated.
>
> Thanks,
> Prashant
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Activemq-Master-dying-without-errors-tp4728716.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>