Dynamic Brokers

[ronchalant <ro...@gmail.com>]

I want to embed an ActiveMQ broker into an application so that when the
application starts, the broker starts.  This broker would be part of a chain
of brokers in a store-forward architecture*, and these brokers may go
up/down.  What I need to be able to do is that as brokers are added or
removed from a system, other brokers recognize this.  All consumers would be
local; this is intended to be used to distribute tasks among different
instances of the application.

Are there samples anywhere of this sort of thing, starting the ActiveMQ
Brokers dynamically within code, and tearing them down within code as well? 
Ideally, in a controlled teardown, I'd like to forward all messages to other
brokers if any are available manually (though this isn't critical, because
if the node comes back up it should just pick those messages up under
store-forward, correct?)

I apologize if I'm missing the examples somewhere on the site, all I'm
seeing is various ways to start/stop brokers, producers and consumers from
the command line.

*tentatively; a master/slave w/ KahaDB might be preferable if it was stable
-- 
View this message in context: http://www.nabble.com/Dynamic-Brokers-tp22689640p22689640.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Dynamic Brokers

[ronchalant <ro...@gmail.com>]

I have fairly easily gotten up a test run of a pair of brokers running in the
same VM, running in a simple store-forward configuration.  

To mimic a fast and slow pair of consumers (mimicking fast and slow workers
in a distributed environment) on one broker I have a queue consumer with a
message limiter sleeping for 50ms between messages, on the other I have a
queue consumer sleeping for 250ms between messages.

When I lower the prefetch limit to something like 3, I get the behavior I'm
looking for from a basic load balancing standpoint (basically the servers
would naturally pull more/less items off the queue as they can work through
them faster/slower; simple "pull" load balancing model).. When sending 2500
text messages, ~1200 chars each, and the "fast" broker processes through 80%
of the messages.. but what I'm also seeing is that the CPU is pegged at near
100% for the duration..  This is the same with or without a prefetch limit
being set..

For my purposes, I don't need to be overly conservative about filesystem
overhead, and the messages we're going to be passing will be relatively
small and simple.. but I'd like to keep memory usage relatively low, and CPU
usage I don't think should be that high.. In the above scenario it does use
about 125MB of memory as well..

What should I be looking to configure?  I have a Derby DB available to me if
it would be better from a CPU/Memory standpoint (performance also isn't
necessarily a concern, I don't expect the depth of any message queues to
extend beyond a few thousand at most).



ronchalant wrote:
> 
> Great, thanks Bruce, I'll look them over.. I don't know how I missed
> them..
> 
> 
> bsnyder wrote:
>> 
>> On Tue, Mar 24, 2009 at 2:58 PM, ronchalant <ro...@gmail.com> wrote:
>>>
>>> I want to embed an ActiveMQ broker into an application so that when the
>>> application starts, the broker starts.  This broker would be part of a
>>> chain
>>> of brokers in a store-forward architecture*, and these brokers may go
>>> up/down.  What I need to be able to do is that as brokers are added or
>>> removed from a system, other brokers recognize this.  All consumers
>>> would be
>>> local; this is intended to be used to distribute tasks among different
>>> instances of the application.
>>>
>>> Are there samples anywhere of this sort of thing, starting the ActiveMQ
>>> Brokers dynamically within code, and tearing them down within code as
>>> well?
>>> Ideally, in a controlled teardown, I'd like to forward all messages to
>>> other
>>> brokers if any are available manually (though this isn't critical,
>>> because
>>> if the node comes back up it should just pick those messages up under
>>> store-forward, correct?)
>> 
>> There are JUnit tests throughout the code base that demonstrate how to
>> do this. Here's some info on some ways to embed a broker:
>> 
>> http://activemq.apache.org/how-do-i-embed-a-broker-inside-a-connection.html
>> 
>>> I apologize if I'm missing the examples somewhere on the site, all I'm
>>> seeing is various ways to start/stop brokers, producers and consumers
>>> from
>>> the command line.
>>>
>>> *tentatively; a master/slave w/ KahaDB might be preferable if it was
>>> stable
>> 
>> Master/slave can be configured either using an activemq.xml and a
>> BrokerFactoryBean or it can be configured completely programmatically
>> using Java code. You'd just need to instantiate and configure the
>> correct Java classes for the persistence of your choice.
>> 
>> Bruce
>> -- 
>> perl -e 'print
>> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
>> );'
>> 
>> Apache ActiveMQ - http://activemq.apache.org/
>> Apache Camel - http://camel.apache.org/
>> Apache ServiceMix - http://servicemix.apache.org/
>> 
>> Blog: http://bruceblog.org/
>> 
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Dynamic-Brokers-tp22689640p22701975.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Dynamic Brokers

[ronchalant <ro...@gmail.com>]

Great, thanks Bruce, I'll look them over.. I don't know how I missed them..


bsnyder wrote:
> 
> On Tue, Mar 24, 2009 at 2:58 PM, ronchalant <ro...@gmail.com> wrote:
>>
>> I want to embed an ActiveMQ broker into an application so that when the
>> application starts, the broker starts.  This broker would be part of a
>> chain
>> of brokers in a store-forward architecture*, and these brokers may go
>> up/down.  What I need to be able to do is that as brokers are added or
>> removed from a system, other brokers recognize this.  All consumers would
>> be
>> local; this is intended to be used to distribute tasks among different
>> instances of the application.
>>
>> Are there samples anywhere of this sort of thing, starting the ActiveMQ
>> Brokers dynamically within code, and tearing them down within code as
>> well?
>> Ideally, in a controlled teardown, I'd like to forward all messages to
>> other
>> brokers if any are available manually (though this isn't critical,
>> because
>> if the node comes back up it should just pick those messages up under
>> store-forward, correct?)
> 
> There are JUnit tests throughout the code base that demonstrate how to
> do this. Here's some info on some ways to embed a broker:
> 
> http://activemq.apache.org/how-do-i-embed-a-broker-inside-a-connection.html
> 
>> I apologize if I'm missing the examples somewhere on the site, all I'm
>> seeing is various ways to start/stop brokers, producers and consumers
>> from
>> the command line.
>>
>> *tentatively; a master/slave w/ KahaDB might be preferable if it was
>> stable
> 
> Master/slave can be configured either using an activemq.xml and a
> BrokerFactoryBean or it can be configured completely programmatically
> using Java code. You'd just need to instantiate and configure the
> correct Java classes for the persistence of your choice.
> 
> Bruce
> -- 
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
> 
> Apache ActiveMQ - http://activemq.apache.org/
> Apache Camel - http://camel.apache.org/
> Apache ServiceMix - http://servicemix.apache.org/
> 
> Blog: http://bruceblog.org/
> 
> 

-- 
View this message in context: http://www.nabble.com/Dynamic-Brokers-tp22689640p22691760.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Dynamic Brokers

[Bruce Snyder <br...@gmail.com>]

On Tue, Mar 24, 2009 at 2:58 PM, ronchalant <ro...@gmail.com> wrote:
>
> I want to embed an ActiveMQ broker into an application so that when the
> application starts, the broker starts.  This broker would be part of a chain
> of brokers in a store-forward architecture*, and these brokers may go
> up/down.  What I need to be able to do is that as brokers are added or
> removed from a system, other brokers recognize this.  All consumers would be
> local; this is intended to be used to distribute tasks among different
> instances of the application.
>
> Are there samples anywhere of this sort of thing, starting the ActiveMQ
> Brokers dynamically within code, and tearing them down within code as well?
> Ideally, in a controlled teardown, I'd like to forward all messages to other
> brokers if any are available manually (though this isn't critical, because
> if the node comes back up it should just pick those messages up under
> store-forward, correct?)

There are JUnit tests throughout the code base that demonstrate how to
do this. Here's some info on some ways to embed a broker:

http://activemq.apache.org/how-do-i-embed-a-broker-inside-a-connection.html

> I apologize if I'm missing the examples somewhere on the site, all I'm
> seeing is various ways to start/stop brokers, producers and consumers from
> the command line.
>
> *tentatively; a master/slave w/ KahaDB might be preferable if it was stable

Master/slave can be configured either using an activemq.xml and a
BrokerFactoryBean or it can be configured completely programmatically
using Java code. You'd just need to instantiate and configure the
correct Java classes for the persistence of your choice.

Bruce
-- 
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'

Apache ActiveMQ - http://activemq.apache.org/
Apache Camel - http://camel.apache.org/
Apache ServiceMix - http://servicemix.apache.org/

Blog: http://bruceblog.org/