Re: Kahadb Activemq holds the lock but does start the broker service or connectors.

[Tim Bain <tb...@alumni.duke.edu>]

Your log file is huge and covers a 5-day period with multiple restarts,
some of which look problem-free. Can you please trim the file down to just
the relevant timeframe?

Also, the stack trace of the main thread appears to be trying to move
expired messages to the DLQ. Could it be that that's just taking a while to
do? Alternatively, could it be that your DLQ setup is wrong in some way,
such as a "loop" that takes messages that expire while in the DLQ and route
them back to the DLQ, where they would expire again, forever? Since you've
not shared your configuration, it's hard to say if this is the problem or
not.

Tim

On Jun 25, 2018 9:11 PM, "Suvp" <su...@nuagenetworks.net> wrote:

Hi,

We had shared file locker with kaha db based persistence adapter in our 3
node master-slave-slave architecture.
Master activemq was restarted couple of time and after a cycle or so it was
seen that even though the activemq  java process is up and it holds the lock
it doesn't start  the broker service and open the transports.

There are no errors in the activemq logs on master nodes.

The other two slave nodes continue to be slaves since the master node
doesn't release the lock.

The query command on activemq doesn't come out and hangs:

[root@vsd-sudhar-set1-node1 bin]# ./activemq query
INFO: Loading '/opt/vsd/amq//bin/env'
INFO: Using java '/usr/java/latest/bin/java'
Java Runtime: Oracle Corporation 1.8.0_172 /usr/java/jre1.8.0_172-amd64
  Heap sizes: current=1005056k  free=994570k  max=1005056k
    JVM args: -Xms1G -Xmx1G
-Djava.util.logging.config.file=logging.properties
-Djava.security.auth.login.config=/opt/vsd/amq/conf/login.config
-Dorg.apache.activemq.SERIALIZABLE_PACKAGES=* -Dvsd.rest.http.retries=5
-Dvsd.rest.http.retry.interval=400
-Dvsd.host.addr=vsd-sudhar-set1-node1.mv.nuagenetworks.net,
vsd-sudhar-set1-node2.mv.nuagenetworks.net,
vsd-sudhar-set1-node3.mv.nuagenetworks.net
-Dactivemq.classpath=/opt/vsd/amq/conf:/opt/vsd/amq//../lib/:
-Dactivemq.home=/opt/vsd/amq -Dactivemq.base=/opt/vsd/amq
-Dactivemq.conf=/opt/vsd/amq/conf -Dactivemq.data=/opt/vsd/amq/data
Extensions classpath:

[/opt/vsd/amq/lib,/opt/vsd/amq/lib/camel,/opt/vsd/amq/lib/optional,/opt/vsd/amq/lib/web,/opt/vsd/amq/lib/extra]
ACTIVEMQ_HOME: /opt/vsd/amq
ACTIVEMQ_BASE: /opt/vsd/amq
ACTIVEMQ_CONF: /opt/vsd/amq/conf
ACTIVEMQ_DATA: /opt/vsd/amq/data
Connecting to JMX URL: service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi

But on slave nodes same command is successful.

Process status:
root      5638     1  0 Jun25 ?        00:00:27 /usr/java/latest/bin/java
-Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties
-Djava.security.auth.login.config=/opt/vsd/amq/conf/login.config
-Dorg.apache.activemq.SERIALIZABLE_PACKAGES=* -Dvsd.rest.http.retries=5
-Dvsd.rest.http.retry.interval=400
-Dvsd.host.addr=vsd-sudhar-set1-node1.mv.nuagenetworks.net,
vsd-sudhar-set1-node2.mv.nuagenetworks.net,
vsd-sudhar-set1-node3.mv.nuagenetworks.net
-Dcom.sun.management.jmxremote -Djava.awt.headless=true
-Djava.io.tmpdir=/opt/vsd/amq/tmp -Xdebug -Xnoagent -Djava.compiler=NONE
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5008
-Dactivemq.classpath=/opt/vsd/amq/conf:/opt/vsd/amq//../lib/:
-Dactivemq.home=/opt/vsd/amq -Dactivemq.base=/opt/vsd/amq
-Dactivemq.conf=/opt/vsd/amq/conf -Dactivemq.data=/opt/vsd/amq/data -jar
/opt/vsd/amq/bin/activemq.jar start

Lock held by same process:
[root@vsd-sudhar-set1-node1 bin]# lsof /opt/vsd/amq/data/database/lock
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF       NODE NAME
java    5638 root  133u   REG   0,44        8 2152124800
/opt/vsd/amq/data/database/lock

I have attached the thread dump and activemq log of the master node.

Can someone help why the broker is not comming up?

threadDump1.txt
<http://activemq.2283324.n4.nabble.com/file/t379043/threadDump1.txt>
activemq.log
<http://activemq.2283324.n4.nabble.com/file/t379043/activemq.log>




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html