You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Arshkr <ph...@unisys.com> on 2019/08/08 07:02:42 UTC
ActiveMQ clustering using NFSv4
Hello,
I'm a newbie to ActiveMQ; I try to configure 2 or 3 VM ActiveMQ nodes
working as Master/slave using NFSv4. I'm reading lots of document, but it
seems I'm more confused about the vast information it offers.
If you don't mind, could you please recommend me a straightforward document
of how to start this HA configuration?
Attached is a simple diagram I try to achieve.
Thank you for your time and help.
Arron.
<http://activemq.2283324.n4.nabble.com/file/t379614/CSIF_-_ActiveMQ_setup.png>
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: ActiveMQ clustering using NFSv4
Posted by Arshkr <ph...@unisys.com>.
Hi François,
Yes, shared NFS file store will be a place for KahaBD directory between the
brokers. According to ActiveMQ, it supports NFS version 4.x; which implies
that I need to configure an additional server running NFS so the three VM
brokers can share a data directory over network.
If I choose this direction, do you know if NFS network shares writing with
some delay?
Another alternative, I think, that I could create a shared disk in ESXi and
each broker will map to this disk, but I'm not sure the file lock will work
properly.
I did a quick test as trying to read and write a text file on the mapped
drive, and both servers can read/write at the same time. Maybe I did
something incorrectly, so your expert recommendations are welcome.
Regards,
Phu Nguyen
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: ActiveMQ clustering using NFSv4
Posted by Arshkr <ph...@unisys.com>.
Hi Tim,
For NFS, I will create a cluster for that later, or I will use another
alternative is add a new LUN on ESXi and create a logical NFS share for all
three brokers to mount for KahaDB.
Would you mind check out my activemq.xml file and help me point out possible
error(s) I have?
Thank you,
Arron
This is how my activemq.xml configured:
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean>
<bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
lazy-init="false" scope="singleton"
init-method="start" destroy-method="stop">
</bean>
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="Broker-03-uuiarray" dataDirectory="C:\activemq\data"
advisorySupport="false" persistent="true" useJmx="true"
deleteAllMessagesOnStartup="false" useShutdownHook="false"
schedulerSupport="false">
<networkConnectors>
<networkConnector
uri="masterslave:(tcp://uis1vcenter2.uismag.com:61616,tcp://uis1dc2.uismag.com:61616,tcp://uuiarray.uismag.com:61616)"/>
</networkConnectors>
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="10000"/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<persistenceAdapter>
<kahaDB directory="B:\Shares\mqdata\data\kahadb"
journalMaxFileLength="32mb"/>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="10 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="5 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire"
rebalanceClusterClients="true" updateClusterClients="true"
updateClusterClientsOnRemove="true"
uri="tcp://0.0.0.0:61616?maximumConnections=10000&wireFormat.maxFrameSize=104857600"
discoveryUri="multicast://default"/>
<transportConnector name="amqp" rebalanceClusterClients="true"
updateClusterClients="true" updateClusterClientsOnRemove="true"
uri="amqp://0.0.0.0:5672?maximumConnections=10000&wireFormat.maxFrameSize=104857600"
discoveryUri="multicast://default"/>
<transportConnector name="stomp" rebalanceClusterClients="true"
updateClusterClients="true" updateClusterClientsOnRemove="true"
uri="stomp://0.0.0.0:61613?maximumConnections=10000&wireFormat.maxFrameSize=104857600"
discoveryUri="multicast://default"/>
<transportConnector name="mqtt" rebalanceClusterClients="true"
updateClusterClients="true" updateClusterClientsOnRemove="true"
uri="mqtt://0.0.0.0:1883?maximumConnections=10000&wireFormat.maxFrameSize=104857600"
discoveryUri="multicast://default"/>
<transportConnector name="ws" rebalanceClusterClients="true"
updateClusterClients="true" updateClusterClientsOnRemove="true"
uri="ws://0.0.0.0:61614?maximumConnections=10000&wireFormat.maxFrameSize=104857600"
discoveryUri="multicast://default"/>
</transportConnectors>
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans"
class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks>
</broker>
<import resource="jetty.xml"/>
</beans>
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: ActiveMQ clustering using NFSv4
Posted by Tim Bain <tb...@alumni.duke.edu>.
So... You have three ActiveMQ nodes that are capable of failing over in the
case of a problem with the current master broker. That's great. But your
NFS server is a single point of failure, and if it goes down, all three
ActiveMQ servers are useless. You might want to figure out how to make your
NFS server be clustered/redundant, to ensure availability in the case of
hardware failure.
Tim
On Thu, Aug 15, 2019, 6:21 PM Arshkr <ph...@unisys.com> wrote:
> Hi Tim,
>
> I think I got it works. Here is my note, in case any newbie like me have
> some idea where to start from :-)
>
> Here is what I did for 3 ActiveMQ servers, and 1 NFS file server:
>
> - Set up an NFS share server on windows 2018, and create a shared folder
> name "activemqdata". Instruction on how to setup:
>
> https://docs.microsoft.com/en-us/windows-server/storage/nfs/deploy-nfs#deploy-nfs-infrastructure
>
> - download ActiveMQ v5.x and copy to 3 Windows 2018 servers, ActiveMQ is
> locate at C:\activemq on each server.
> Instruction on how to setup:
> https://wiki.eveoh.nl/pages/viewpage.action?pageId=14287030
>
> - From each ActiveMQ server, map a network drive activemqdata\ as a local
> drive and assigned a drive letter. In this case, I use B:\activemqdata
>
> - move a data\ directory in ActiveMQ installation to B:\activemq\data, and
> deleted remaining "data\" on last two servers
>
> - modify activemq.xml in conf\ directory to use kahadb in
> "B:\activemq\data"
>
> - Start ActiveMQ on each server, and everything works as expected. In my
> case, I have 3 ActiveMQ servers and an NFS file server.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: ActiveMQ clustering using NFSv4
Posted by Arshkr <ph...@unisys.com>.
Hi Tim,
I think I got it works. Here is my note, in case any newbie like me have
some idea where to start from :-)
Here is what I did for 3 ActiveMQ servers, and 1 NFS file server:
- Set up an NFS share server on windows 2018, and create a shared folder
name "activemqdata". Instruction on how to setup:
https://docs.microsoft.com/en-us/windows-server/storage/nfs/deploy-nfs#deploy-nfs-infrastructure
- download ActiveMQ v5.x and copy to 3 Windows 2018 servers, ActiveMQ is
locate at C:\activemq on each server.
Instruction on how to setup:
https://wiki.eveoh.nl/pages/viewpage.action?pageId=14287030
- From each ActiveMQ server, map a network drive activemqdata\ as a local
drive and assigned a drive letter. In this case, I use B:\activemqdata
- move a data\ directory in ActiveMQ installation to B:\activemq\data, and
deleted remaining "data\" on last two servers
- modify activemq.xml in conf\ directory to use kahadb in "B:\activemq\data"
- Start ActiveMQ on each server, and everything works as expected. In my
case, I have 3 ActiveMQ servers and an NFS file server.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: ActiveMQ clustering using NFSv4
Posted by Tim Bain <tb...@alumni.duke.edu>.
Phu,
You can use any shared filesystem for storing the data, but to act as the
shared lock, the filesystem must support specific lock operations. NFSv4
supports those options. NFSv3 and before does not. Other filesystems may or
may not, though your experiment makes it sound like ESXi may not.
However, via the pluggable storage locker feature (
https://activemq.apache.org/pluggable-storage-lockers), you can use a
different technology for the shared lock than you use for the data. So you
could store your data in EXSi but use a JDBC database for the lock, for
example.
Tim
On Mon, Aug 12, 2019, 6:13 PM Arshkr <ph...@unisys.com> wrote:
> Hi Tim,
>
> Thank you for your reply.
> Yes, I have read this document before, and master/slave concept is clear to
> me. My big question is on NFS shared file system where ActiveMQ is writing
> data on.
> I have only one option for this setup is to use NFS shared file system for
> the data directory. I don't know if NFSv4 is the only option or there is
> another alternative such as local file share storage vs. network file share
> or not.
>
>
> Regards,
> Phu Nguyen.
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: ActiveMQ clustering using NFSv4
Posted by Arshkr <ph...@unisys.com>.
Hi Tim,
Thank you for your reply.
Yes, I have read this document before, and master/slave concept is clear to
me. My big question is on NFS shared file system where ActiveMQ is writing
data on.
I have only one option for this setup is to use NFS shared file system for
the data directory. I don't know if NFSv4 is the only option or there is
another alternative such as local file share storage vs. network file share
or not.
Regards,
Phu Nguyen.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: ActiveMQ clustering using NFSv4
Posted by Tim Bain <tb...@alumni.duke.edu>.
Have you read the official documentation at
https://activemq.apache.org/shared-file-system-master-slave? Does it
answer your questions?
If not, if you post specific questions about specific things you don't
understand to this list, hopefully we can help you work through anything
that's not clear. And if you get to the end and say, "Wow, if only that
page had told me X, I'd have gotten it right away," we can add useful
content to that page.
Tim
On Thu, Aug 8, 2019, 8:18 AM Francois Papon <fr...@openobject.fr>
wrote:
> Hi,
>
> Are you using NFS to share the KahaBD directory between the brokers?
>
> regards,
>
> François
> fpapon@apache.org
>
> Le 08/08/2019 à 09:02, Arshkr a écrit :
> > Hello,
> >
> > I'm a newbie to ActiveMQ; I try to configure 2 or 3 VM ActiveMQ nodes
> > working as Master/slave using NFSv4. I'm reading lots of document, but it
> > seems I'm more confused about the vast information it offers.
> >
> > If you don't mind, could you please recommend me a straightforward
> document
> > of how to start this HA configuration?
> >
> > Attached is a simple diagram I try to achieve.
> >
> > Thank you for your time and help.
> >
> > Arron.
> > <
> http://activemq.2283324.n4.nabble.com/file/t379614/CSIF_-_ActiveMQ_setup.png>
>
> >
> >
> >
> > --
> > Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: ActiveMQ clustering using NFSv4
Posted by Francois Papon <fr...@openobject.fr>.
Hi,
Are you using NFS to share the KahaBD directory between the brokers?
regards,
François
fpapon@apache.org
Le 08/08/2019 à 09:02, Arshkr a écrit :
> Hello,
>
> I'm a newbie to ActiveMQ; I try to configure 2 or 3 VM ActiveMQ nodes
> working as Master/slave using NFSv4. I'm reading lots of document, but it
> seems I'm more confused about the vast information it offers.
>
> If you don't mind, could you please recommend me a straightforward document
> of how to start this HA configuration?
>
> Attached is a simple diagram I try to achieve.
>
> Thank you for your time and help.
>
> Arron.
> <http://activemq.2283324.n4.nabble.com/file/t379614/CSIF_-_ActiveMQ_setup.png>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html