You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Erik Drolshammer <er...@fjas.no> on 2009/03/23 17:00:53 UTC
Clustering for HA
Hi!
We try to set up two brokers according to [1] to get some redundancy in
our solution. The shared filesystem is based on GFS, but it doesn't seem
to work that well. The master node use a lot of cpu, but the throughput
is horrible.
Can anyone point me to some resources describing the setup?
Any debug tips?
[1] http://activemq.apache.org/shared-file-system-master-slave.html
--
Best regards,
Erik Drolshammer
Re: Clustering for HA
Posted by Bruce Snyder <br...@gmail.com>.
On Wed, Mar 25, 2009 at 1:19 AM, Erik Drolshammer <er...@fjas.no> wrote:
> I don't want to use JDBC if I can avoid it. I thought I _had_ to after
> reading http://activemq.apache.org/shared-file-system-master-slave.html.
>
> When I removed the JDBC-element and thus used KahaDB things seems to be
> working again. Strange that the derbydb didn't work at all. (One message
> dispatch per minute instead of 500 per second or so normally.)
This is common when Derby is run inside the same JVM as ActiveMQ
because Derby is very hard on the garbage collector. We recommend
splitting Derby into it's own JVM if you want to use it.
> Looks promising. Any idea when it will be released?
The AMQ message store was released when ActiveMQ 5.0 came out and it
is the default persistent mechanism in the 5.x version. It makes use
of Kaha as the message reference store. I suggest trying the AMQ
message store as it is incredibly fast:
http://activemq.apache.org/amq-message-store.html
> Thanks for you help. :)
>
> FYI;
> When we have verified our setup properly I want to create a jira issue
> suggesting documentation extensions. Until then, I use
> http://wiki.community.objectware.no/display/smidigtonull/ActiveMQ+-+clustering+for+High+Availability
> as scratch area.
Great, let us know if there's anything you'd like to contribute to the
documentation.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.apache.org/
Apache Camel - http://camel.apache.org/
Apache ServiceMix - http://servicemix.apache.org/
Blog: http://bruceblog.org/
Re: Clustering for HA
Posted by Erik Drolshammer <er...@fjas.no>.
Bruce Snyder wrote:
> On Tue, Mar 24, 2009 at 1:17 AM, Erik Drolshammer <er...@fjas.no> wrote:
>> Bruce Snyder wrote:
>>> On Mon, Mar 23, 2009 at 10:00 AM, Erik Drolshammer <er...@fjas.no>
>>> wrote:
>> It seems this setup use DerbyDB and not Kahastore. Is this the only option?
>> Are there alternative setups that I might try?
>
> Yes, the journaledJDBC element will use Apache Derby by default, but
> you can specify a bean to use as the datasource in the activemq.xml so
> that you can use whatever database you prefer.
>
> Other persistence options include non-journaled JDBC as noted here:
>
> http://activemq.apache.org/persistence.html#Persistence-JDBCPersistencewithoutJournaling
>
> If you're using JDBC-based persistence, then you should look at using
> JDBC master/slave:
>
> http://activemq.apache.org/jdbc-master-slave.html
>
> Another persistence option is Kaha persistence as described here:
>
> http://activemq.apache.org/persistence.html#Persistence-KahaPersistence
>
> With this style of persistence, you should use the KahaDB master/slave:
>
> http://activemq.apache.org/kahadb-master-slave.html
I don't want to use JDBC if I can avoid it. I thought I _had_ to after
reading http://activemq.apache.org/shared-file-system-master-slave.html.
When I removed the JDBC-element and thus used KahaDB things seems to be
working again. Strange that the derbydb didn't work at all. (One message
dispatch per minute instead of 500 per second or so normally.)
> This is the default persistence mechanism in ActiveMQ 5.x. If you're
> using this style of persistence, then you should be using the shared
> filesystem master/slave:
>
> http://activemq.apache.org/shared-file-system-master-slave.html
Looks promising. Any idea when it will be released?
Thanks for you help. :)
FYI;
When we have verified our setup properly I want to create a jira issue
suggesting documentation extensions. Until then, I use
http://wiki.community.objectware.no/display/smidigtonull/ActiveMQ+-+clustering+for+High+Availability
as scratch area.
--
Best regards,
Erik Drolshammer
Re: Clustering for HA
Posted by Bruce Snyder <br...@gmail.com>.
On Tue, Mar 24, 2009 at 1:17 AM, Erik Drolshammer <er...@fjas.no> wrote:
> Bruce Snyder wrote:
>>
>> On Mon, Mar 23, 2009 at 10:00 AM, Erik Drolshammer <er...@fjas.no>
>> wrote:
>>>
>>> Hi!
>
> Good morning :)
>
>
>>> We try to set up two brokers according to [1] to get some redundancy in
>>> our
>>> solution. The shared filesystem is based on GFS, but it doesn't seem to
>>> work
>>> that well. The master node use a lot of cpu, but the throughput is
>>> horrible.
>>>
>>>
>>> Can anyone point me to some resources describing the setup?
>>> Any debug tips?
>>
>> The only setup that's needed for the shared filesystem master/slave is
>> noting the same data directory location for each ActiveMQ instance.
>> That's it. ActiveMQ takes care of the rest based on the filesystem
>> locking.
>
> I have this element in activemq.conf on two nodes:
>
> <broker xmlns="http://activemq.apache.org/schema/core" useJmx="true"
> dataDirectory="${activemq.base}/data">
>
> <persistenceAdapter>
> <journaledJDBC dataDirectory="/home/mq"/>
> </persistenceAdapter>
>
> Does this seem correct (and adequate)?
>
> It seems this setup use DerbyDB and not Kahastore. Is this the only option?
> Are there alternative setups that I might try?
Yes, the journaledJDBC element will use Apache Derby by default, but
you can specify a bean to use as the datasource in the activemq.xml so
that you can use whatever database you prefer.
Other persistence options include non-journaled JDBC as noted here:
http://activemq.apache.org/persistence.html#Persistence-JDBCPersistencewithoutJournaling
If you're using JDBC-based persistence, then you should look at using
JDBC master/slave:
http://activemq.apache.org/jdbc-master-slave.html
Another persistence option is Kaha persistence as described here:
http://activemq.apache.org/persistence.html#Persistence-KahaPersistence
With this style of persistence, you should use the KahaDB master/slave:
http://activemq.apache.org/kahadb-master-slave.html
The last persistence option is to use the AMQ Message Store as described here:
http://activemq.apache.org/amq-message-store.html
This is the default persistence mechanism in ActiveMQ 5.x. If you're
using this style of persistence, then you should be using the shared
filesystem master/slave:
http://activemq.apache.org/shared-file-system-master-slave.html
>> According to the Wikipedia entry for the Google File System (GFS), it
>> is mainly used for data that is 'extremely rarely overwritten, or
>> shrunk; files are usually appended to or read.'
>>
>> http://en.wikipedia.org/wiki/Google_File_System
>>
>> Given this essential fact, I'd say that GFS is probably not a good
>> candidate for use by ActiveMQ since the ActiveMQ data is in almost a
>> constant churn of being written and removed as messages flow through
>> the broker. Could this be the cause of the high CPU usage and poor
>> throughput?
>
> Perhaps. Btw, I meant Redhat GFS [1], not Google FS. Do you know if Redhat
> FS is a good solution?
> Or what setup would you recommend?
>
> We currently use ActiveMQ 5.2.0 on RedHat platform as indicated.
Well, in reading about RedHat GFS it is a distributed, journaled
cluster filesystem and volume manager. In the quick amount of research
I performed I couldn't find any info about performance problems. But
just like any sophisticated filesystem and volume manager, tuning can
be a rather complex task requiring someone with intimate knowledge of
the filesystem and volume manager.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.apache.org/
Apache Camel - http://camel.apache.org/
Apache ServiceMix - http://servicemix.apache.org/
Blog: http://bruceblog.org/
Re: Clustering for HA
Posted by Erik Drolshammer <er...@fjas.no>.
Bruce Snyder wrote:
> On Mon, Mar 23, 2009 at 10:00 AM, Erik Drolshammer <er...@fjas.no> wrote:
>> Hi!
Good morning :)
>> We try to set up two brokers according to [1] to get some redundancy in our
>> solution. The shared filesystem is based on GFS, but it doesn't seem to work
>> that well. The master node use a lot of cpu, but the throughput is horrible.
>>
>>
>> Can anyone point me to some resources describing the setup?
>> Any debug tips?
>
> The only setup that's needed for the shared filesystem master/slave is
> noting the same data directory location for each ActiveMQ instance.
> That's it. ActiveMQ takes care of the rest based on the filesystem
> locking.
I have this element in activemq.conf on two nodes:
<broker xmlns="http://activemq.apache.org/schema/core" useJmx="true"
dataDirectory="${activemq.base}/data">
<persistenceAdapter>
<journaledJDBC dataDirectory="/home/mq"/>
</persistenceAdapter>
Does this seem correct (and adequate)?
It seems this setup use DerbyDB and not Kahastore. Is this the only
option? Are there alternative setups that I might try?
> According to the Wikipedia entry for the Google File System (GFS), it
> is mainly used for data that is 'extremely rarely overwritten, or
> shrunk; files are usually appended to or read.'
>
> http://en.wikipedia.org/wiki/Google_File_System
>
> Given this essential fact, I'd say that GFS is probably not a good
> candidate for use by ActiveMQ since the ActiveMQ data is in almost a
> constant churn of being written and removed as messages flow through
> the broker. Could this be the cause of the high CPU usage and poor
> throughput?
Perhaps. Btw, I meant Redhat GFS [1], not Google FS. Do you know if
Redhat FS is a good solution?
Or what setup would you recommend?
We currently use ActiveMQ 5.2.0 on RedHat platform as indicated.
[1]http://www.redhat.com/gfs/
--
Best regards,
Erik Drolshammer
Re: Clustering for HA
Posted by Bruce Snyder <br...@gmail.com>.
On Mon, Mar 23, 2009 at 10:00 AM, Erik Drolshammer <er...@fjas.no> wrote:
> Hi!
> We try to set up two brokers according to [1] to get some redundancy in our
> solution. The shared filesystem is based on GFS, but it doesn't seem to work
> that well. The master node use a lot of cpu, but the throughput is horrible.
>
>
> Can anyone point me to some resources describing the setup?
> Any debug tips?
The only setup that's needed for the shared filesystem master/slave is
noting the same data directory location for each ActiveMQ instance.
That's it. ActiveMQ takes care of the rest based on the filesystem
locking.
According to the Wikipedia entry for the Google File System (GFS), it
is mainly used for data that is 'extremely rarely overwritten, or
shrunk; files are usually appended to or read.'
http://en.wikipedia.org/wiki/Google_File_System
Given this essential fact, I'd say that GFS is probably not a good
candidate for use by ActiveMQ since the ActiveMQ data is in almost a
constant churn of being written and removed as messages flow through
the broker. Could this be the cause of the high CPU usage and poor
throughput?
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.apache.org/
Apache Camel - http://camel.apache.org/
Apache ServiceMix - http://servicemix.apache.org/
Blog: http://bruceblog.org/