You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Suvp <su...@nuagenetworks.net> on 2018/05/07 23:42:44 UTC
Kahadb with leased database locker broker not closing connector
when connection to NFS fails.
Hi,
In our activemq installation we are using kahaDb as the persistence adapter
with leased-database-locker for locking mechanism. The shared file system
used is NFSv3.
We wanted to test scenario where the Master node has access to mysql through
which it will remain master by acquiring lock but looses connection to
shared file system or NFS and see that the master switches since its not
able to contact NFS.
What i observed was in the activemq logs there are no exceptions when active
broker looses connection to NFS and the it still remains as master and the
clients fail to receive messages when it looses connection.
Following is the Exception handler used:
<ioExceptionHandler>
<leaseLockerIOExceptionHandler ignoreSQLExceptions="false"
stopStartConnectors="true"/>
</ioExceptionHandler>
And following is the persistence adapter configuration:
<persistenceAdapter>
<kahaDB directory="/home/sunil" lockKeepAlivePeriod="5000">
<locker>
<lease-database-locker lockAcquireSleepInterval="10000"
dataSource="#mysql-ds">
<statements>
<statements lockTableName="ACTIVEMQ_LOCK"/>
</statements>
</lease-database-locker>
</locker>
</kahaDB>
</persistenceAdapter>
And the NFS configuration:
10.31.103.147:/home/sunil 20G 103M 20G 1% /home/sunil
/home/sunil 10.31.52.55(rw,sync,fsid=0) 10.31.52.56(rw,sync,fsid=0)
10.31.52.57(rw,sync,fsid=0)
From the NFS server using iptable rules the master node is dropped:
10.31.52.56 is master node
iptables -I OUTPUT -s 10.31.52.56 -j DROP
iptables -I INPUT -s 10.31.52.56 -j DROP
service iptables save
Does the activemq raise IOException when it looses connection to Shared
Filesystem ? And i can use NFSv3 as the shared file system rite ?
Thanks in advance
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Kahadb with leased database locker broker not closing connector
when connection to NFS fails.
Posted by Tim Bain <tb...@alumni.duke.edu>.
Most of your questions are beyond my knowledge of NFS, so I'm not going to
be able to spot problems in your config or warn you about potential
problems. Maybe someone else can.
For NFSv3, the only concern I've heard about it in the context of ActiveMQ
is that its locking mechanisms don't allow it to be used for a filesystem
locker. Since you're not doing that, I think you'd be OK using it for the
data files themselves, but again, this is not an area with which I have any
personal experience.
Tim
Re: Kahadb with leased database locker broker not closing connector
when connection to NFS fails.
Posted by Suvp <su...@nuagenetworks.net>.
Hi Tim,
Thanks for the reply. I had figured out the problem , it was due to options
used in mount which was causing issue .
I had used default options of timeo and retrans with Hard mode. After
changing the timeo and retrans with below values it started to work fine.
The broker was able to identify the network failure.
> nfsstat -m
/home/sunil from 10.31.103.147:/amq
Flags:
rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,port=0,timeo=1,retrans=2,sec=sys,clientaddr=10.31.52.55,local_lock=none,addr=10.31.103.147
Please let me know if you foresee any further issues with about mount
options.
Also wanted to know if any specific server settings has to be done on NFS
server?
Another question was can i use NFSv3?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Kahadb with leased database locker broker not closing connector
when connection to NFS fails.
Posted by Tim Bain <tb...@alumni.duke.edu>.
I'm not an expert on NFS, but I'd have expected that the network
connectivity failure to the NFS share to be detected when the broker
attempts to read or write the file, not just when you disable connections
to it. Your description implies that you're doing the latter; what is the
behavior of the broker when you do the former, e.g. by attempting to
publish a message to the broker after you disable the network connectivity
to the share?
Tim
On Mon, May 7, 2018 at 5:42 PM, Suvp <su...@nuagenetworks.net> wrote:
> Hi,
>
> In our activemq installation we are using kahaDb as the persistence adapter
> with leased-database-locker for locking mechanism. The shared file system
> used is NFSv3.
>
> We wanted to test scenario where the Master node has access to mysql
> through
> which it will remain master by acquiring lock but looses connection to
> shared file system or NFS and see that the master switches since its not
> able to contact NFS.
>
> What i observed was in the activemq logs there are no exceptions when
> active
> broker looses connection to NFS and the it still remains as master and the
> clients fail to receive messages when it looses connection.
>
> Following is the Exception handler used:
> <ioExceptionHandler>
> <leaseLockerIOExceptionHandler ignoreSQLExceptions="false"
> stopStartConnectors="true"/>
> </ioExceptionHandler>
>
> And following is the persistence adapter configuration:
> <persistenceAdapter>
> <kahaDB directory="/home/sunil" lockKeepAlivePeriod="5000">
> <locker>
>
> <lease-database-locker lockAcquireSleepInterval="10000"
> dataSource="#mysql-ds">
> <statements>
>
> <statements lockTableName="ACTIVEMQ_LOCK"/>
> </statements>
> </lease-database-locker>
> </locker>
> </kahaDB>
> </persistenceAdapter>
>
>
> And the NFS configuration:
> 10.31.103.147:/home/sunil 20G 103M 20G 1% /home/sunil
>
> /home/sunil 10.31.52.55(rw,sync,fsid=0) 10.31.52.56(rw,sync,fsid=0)
> 10.31.52.57(rw,sync,fsid=0)
>
> From the NFS server using iptable rules the master node is dropped:
> 10.31.52.56 is master node
> iptables -I OUTPUT -s 10.31.52.56 -j DROP
> iptables -I INPUT -s 10.31.52.56 -j DROP
> service iptables save
>
> Does the activemq raise IOException when it looses connection to Shared
> Filesystem ? And i can use NFSv3 as the shared file system rite ?
>
> Thanks in advance
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>