Topology Question

[Rob Bugh <ro...@earthlink.net>]

Hello All,
   Is this the forum to ask How-To questons with regards to deploying
ActiveMQ in a given topology? I'm new to ActiveMQ and could use a bit of
advice. If this kind of question should be posted elsewhere, my apologies,
please point me there.

My topology
--------------
In a nutshell, I have several datacenters that need certain data to be kept
in sync. One datacenter is on the west coast, one is on the east coast, and
another is in Europe. Each datacenter has its own database, but one table in
each of the databases needs to be kept in sync with each other. Each
datacenter consists of several Tomcat servers fronted by a load balancer and
a database server.

I've read through the documentation and it seems that a network of brokers
may be what I need, but I would like to hear from others that have similar
topologies. Where things get fuzzy for me is trying to determine exactly
where the brokers will live? My first thought was to put a broker in each
Tomcat instance, running as a resource. This seems like a good choice since
it provides broker fail over, but how do you, for instance, point the
brokers at the west coast datacenter to the ones at the other datacenters?
Ideally, I want each datacenter to think of the other datacenters as having
a single broker instead of several embedded into the Tomcat instances. Can
one broker connect to another through a load balancer?

Applications running on the Tomcat instances that wants to update the table
in the database would publish to a topic. This would allow the message to be
seen by all of the datacenters. Topic listeners running as a resource in the
Tomcat instances would receive the message and update the database. The
problem with this is, since a topic is used, all listeners will receive the
message, correct? Meaning, the database will be updated multiple times, once
per listeners in a single datacenter. Ideally, I want the database to be
updated once per message.

What are some other ways to use ActiveMQ to solve this problem?

Regards,
Rob




-- 
View this message in context: http://www.nabble.com/Topology-Question-tf3231705s2354.html#a8979342
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Topology Question

[Juan Jose Comellas <jc...@gmail.com>]

PostgreSQL does have several replication solutions. The one I'm using
is Slony-1, which works very well and is available at:
http://www.slony.info.


On 2/17/07, Rob Bugh <ro...@earthlink.net> wrote:
>
> Thanks for the input, James.
>
> I'm using Postgresql so I don't think there is a DB vendor solution.
>
> Are you suggesting to create different topics for each datacenter or share
> the same topic across all the datacenters?
>
> One of the issues I'm having is exactly where the brokers will be at each
> datacenter. I don't have extra hardware for the brokers so they'll have to
> live on the Tomcat servers. My first thought was to make them run as
> resources within the Tomcat instances. Are you aware of any issues setting
> up a Master/Slave configuration this way?
>
> I forgot to mention previously that the number of datacenters may grow in
> the future so it would be nice if the solution required a minimum of changes
> to add a new datacenter.
>
> Regards,
> Rob
>
>
> James.Strachan wrote:
> >
> > On 2/15/07, Rob Bugh <ro...@...t> wrote:
> >>
> >> Hello All,
> >>    Is this the forum to ask How-To questons with regards to deploying
> >> ActiveMQ in a given topology?
> >
> > Sure!
> >
> >> My topology
> >> --------------
> >> In a nutshell, I have several datacenters that need certain data to be
> >> kept
> >> in sync. One datacenter is on the west coast, one is on the east coast,
> >> and
> >> another is in Europe. Each datacenter has its own database, but one table
> >> in
> >> each of the databases needs to be kept in sync with each other. Each
> >> datacenter consists of several Tomcat servers fronted by a load balancer
> >> and
> >> a database server.
> >
> > BTW if all you want is to do database replication, it might be your
> > database vendor has a solution?
> >
> >
> >> I've read through the documentation and it seems that a network of
> >> brokers
> >> may be what I need, but I would like to hear from others that have
> >> similar
> >> topologies. Where things get fuzzy for me is trying to determine exactly
> >> where the brokers will live? My first thought was to put a broker in each
> >> Tomcat instance, running as a resource. This seems like a good choice
> >> since
> >> it provides broker fail over, but how do you, for instance, point the
> >> brokers at the west coast datacenter to the ones at the other
> >> datacenters?
> >> Ideally, I want each datacenter to think of the other datacenters as
> >> having
> >> a single broker instead of several embedded into the Tomcat instances.
> >> Can
> >> one broker connect to another through a load balancer?
> >>
> >> Applications running on the Tomcat instances that wants to update the
> >> table
> >> in the database would publish to a topic. This would allow the message to
> >> be
> >> seen by all of the datacenters. Topic listeners running as a resource in
> >> the
> >> Tomcat instances would receive the message and update the database. The
> >> problem with this is, since a topic is used, all listeners will receive
> >> the
> >> message, correct? Meaning, the database will be updated multiple times,
> >> once
> >> per listeners in a single datacenter. Ideally, I want the database to be
> >> updated once per message.
> >>
> >> What are some other ways to use ActiveMQ to solve this problem?
> >
> > OK here would be my recommendations...
> >
> > Run one or two brokers per data centres using JDBC master/slave to the
> > local DB.
> >
> > Then you can create a durable topic subscription on each data centre
> > to listen for its own set of updates from ther others ; then you can
> > either leave the brokers separate and have multiple consumers in each
> > data centre which connect to each data centre using failover: or you
> > can use a network to store-and-forward messages from broker-to-broker.
> >
> > --
> >
> > James
> > -------
> > http://radio.weblogs.com/0112098/
> >
> >
>
> --
> View this message in context: http://www.nabble.com/Topology-Question-tf3231705s2354.html#a9020160
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>

Re: Topology Question

[James Strachan <ja...@gmail.com>]

On 2/17/07, Rob Bugh <ro...@earthlink.net> wrote:
>
> Thanks for the input, James.
>
> I'm using Postgresql so I don't think there is a DB vendor solution.
>
> Are you suggesting to create different topics for each datacenter or share
> the same topic across all the datacenters?

You could use either really; the important thing is that each data
centres's messages get sent to every other data centre.


> One of the issues I'm having is exactly where the brokers will be at each
> datacenter. I don't have extra hardware for the brokers so they'll have to
> live on the Tomcat servers. My first thought was to make them run as
> resources within the Tomcat instances. Are you aware of any issues setting
> up a Master/Slave configuration this way?

No - you can run them anywhere. Having a master/slave broker inside
each Tomcat is a good idea. Then if the Tomcat running the master
dies, one of the slaves will take over.


> I forgot to mention previously that the number of datacenters may grow in
> the future so it would be nice if the solution required a minimum of changes
> to add a new datacenter.

Setting up a network of brokers is pretty straightforward, its only a
single URL that changes in the config as you add data-centres.

-- 

James
-------
http://radio.weblogs.com/0112098/

Re: Topology Question

[Rob Bugh <ro...@earthlink.net>]

Thanks for the input, James.

I'm using Postgresql so I don't think there is a DB vendor solution.

Are you suggesting to create different topics for each datacenter or share
the same topic across all the datacenters?

One of the issues I'm having is exactly where the brokers will be at each
datacenter. I don't have extra hardware for the brokers so they'll have to
live on the Tomcat servers. My first thought was to make them run as
resources within the Tomcat instances. Are you aware of any issues setting
up a Master/Slave configuration this way?

I forgot to mention previously that the number of datacenters may grow in
the future so it would be nice if the solution required a minimum of changes
to add a new datacenter.

Regards,
Rob


James.Strachan wrote:
> 
> On 2/15/07, Rob Bugh <ro...@...t> wrote:
>>
>> Hello All,
>>    Is this the forum to ask How-To questons with regards to deploying
>> ActiveMQ in a given topology?
> 
> Sure!
> 
>> My topology
>> --------------
>> In a nutshell, I have several datacenters that need certain data to be
>> kept
>> in sync. One datacenter is on the west coast, one is on the east coast,
>> and
>> another is in Europe. Each datacenter has its own database, but one table
>> in
>> each of the databases needs to be kept in sync with each other. Each
>> datacenter consists of several Tomcat servers fronted by a load balancer
>> and
>> a database server.
> 
> BTW if all you want is to do database replication, it might be your
> database vendor has a solution?
> 
> 
>> I've read through the documentation and it seems that a network of
>> brokers
>> may be what I need, but I would like to hear from others that have
>> similar
>> topologies. Where things get fuzzy for me is trying to determine exactly
>> where the brokers will live? My first thought was to put a broker in each
>> Tomcat instance, running as a resource. This seems like a good choice
>> since
>> it provides broker fail over, but how do you, for instance, point the
>> brokers at the west coast datacenter to the ones at the other
>> datacenters?
>> Ideally, I want each datacenter to think of the other datacenters as
>> having
>> a single broker instead of several embedded into the Tomcat instances.
>> Can
>> one broker connect to another through a load balancer?
>>
>> Applications running on the Tomcat instances that wants to update the
>> table
>> in the database would publish to a topic. This would allow the message to
>> be
>> seen by all of the datacenters. Topic listeners running as a resource in
>> the
>> Tomcat instances would receive the message and update the database. The
>> problem with this is, since a topic is used, all listeners will receive
>> the
>> message, correct? Meaning, the database will be updated multiple times,
>> once
>> per listeners in a single datacenter. Ideally, I want the database to be
>> updated once per message.
>>
>> What are some other ways to use ActiveMQ to solve this problem?
> 
> OK here would be my recommendations...
> 
> Run one or two brokers per data centres using JDBC master/slave to the
> local DB.
> 
> Then you can create a durable topic subscription on each data centre
> to listen for its own set of updates from ther others ; then you can
> either leave the brokers separate and have multiple consumers in each
> data centre which connect to each data centre using failover: or you
> can use a network to store-and-forward messages from broker-to-broker.
> 
> -- 
> 
> James
> -------
> http://radio.weblogs.com/0112098/
> 
> 

-- 
View this message in context: http://www.nabble.com/Topology-Question-tf3231705s2354.html#a9020160
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Topology Question

[James Strachan <ja...@gmail.com>]

On 2/15/07, Rob Bugh <ro...@earthlink.net> wrote:
>
> Hello All,
>    Is this the forum to ask How-To questons with regards to deploying
> ActiveMQ in a given topology?

Sure!

> My topology
> --------------
> In a nutshell, I have several datacenters that need certain data to be kept
> in sync. One datacenter is on the west coast, one is on the east coast, and
> another is in Europe. Each datacenter has its own database, but one table in
> each of the databases needs to be kept in sync with each other. Each
> datacenter consists of several Tomcat servers fronted by a load balancer and
> a database server.

BTW if all you want is to do database replication, it might be your
database vendor has a solution?


> I've read through the documentation and it seems that a network of brokers
> may be what I need, but I would like to hear from others that have similar
> topologies. Where things get fuzzy for me is trying to determine exactly
> where the brokers will live? My first thought was to put a broker in each
> Tomcat instance, running as a resource. This seems like a good choice since
> it provides broker fail over, but how do you, for instance, point the
> brokers at the west coast datacenter to the ones at the other datacenters?
> Ideally, I want each datacenter to think of the other datacenters as having
> a single broker instead of several embedded into the Tomcat instances. Can
> one broker connect to another through a load balancer?
>
> Applications running on the Tomcat instances that wants to update the table
> in the database would publish to a topic. This would allow the message to be
> seen by all of the datacenters. Topic listeners running as a resource in the
> Tomcat instances would receive the message and update the database. The
> problem with this is, since a topic is used, all listeners will receive the
> message, correct? Meaning, the database will be updated multiple times, once
> per listeners in a single datacenter. Ideally, I want the database to be
> updated once per message.
>
> What are some other ways to use ActiveMQ to solve this problem?

OK here would be my recommendations...

Run one or two brokers per data centres using JDBC master/slave to the
local DB.

Then you can create a durable topic subscription on each data centre
to listen for its own set of updates from ther others ; then you can
either leave the brokers separate and have multiple consumers in each
data centre which connect to each data centre using failover: or you
can use a network to store-and-forward messages from broker-to-broker.

-- 

James
-------
http://radio.weblogs.com/0112098/