ActiveMQ 6.0 Broker Core Prototype

[Colin MacNaughton <cm...@progress.com>]

Hi Everyone,

So over the last couple of months Hiram and I have been doing some
experimenting with some bits that we hope might start to serve as a
foundation for a new broker core for 6.0. Now that we're kind of out of
the investigation phase it seems like a good time to start some
discussion on what we've been looking at. The primary goals that we had
in mind were:

-Improved Scalability (particularly in threading model)
-Improved Performance.
-Improved Memory Management / Flow Control model
-Improved/Simplified Queue and Cursored Queue.
-Flexible protocol handler support. 
-Better modularization of components. 
So with that in mind, over the next couple of days/weeks I though we
could get a couple of threads going around the above points. First off
to get started looking at some of the stuff:
It is located in the activemq sandbox at:
https://svn.apache.org/repos/asf/activemq/sandbox/activemq-flow
<https://svn.apache.org/repos/asf/activemq/sandbox/activemq-flow%7D> 
Architecture Notes
For a detailed list of architecture notes, check out the webgen
directory, the README file there gives instructions on how to generate
architecture notes. This also
Running some test cases:
There are few test cases that should help give you a feel for some of
this stuff.
When running these test cases you should be sure to use the -server jvm
argument:
1.	activemq-queue module org.apache.activemq.flow.MockBrokerTest.
This is a good starting point for looking at some of the flow control,
dispatcher and queue modules. It is a very basic MockBroker which uses a
lightweight proto-buf based protocol. You should feel free to play
around with some of the hardcoded options in the test (e.g. ptp=true,
tcp=true, threadsPerDispatcher etc).
2.	activemq-openwire module
org.apache.activemq.broker.SharedQueuePerfTest. This test does
performance testing against the SharedQueue implementation and is a good
place to start looking at queue persistence and the CursoredQueue. It
employs openwire messages with using network connecitons. Check out
org.apache.activemq.broker.BrokerQueueStore which currently hardcodes
the queue sizes; you can change DEFAULT_SHARED_QUEUE_PAGING_THRESHOLD
and the associated persistence policy to play around with paging.
3.	activemq-openwire module
org.apache.activemq.broker.openwire.OpenWireBrokerTest. This test uses
the full blown broker, adds in a tcp transport layert and provides
options for persistence and durable subscriptions (albeit asynchronous
at the moment). It extends org.apache.activemq.broker.BrokerTest in the
activemq-broker module which has some additional hard coded options to
play around with. Again refer to
org.apache.activemq.broker.BrokerQueueStore to mess around with queue
sizes.
It is still a work in progress, so any input would be a tremendous help!
Cheers,
Colin

RE: ActiveMQ 6.0 Broker Core Prototype

[Colin MacNaughton <co...@gmail.com>]

Hi Everyone,

Here's a resend with some better formatting:

So over the last couple of months Hiram and I have been doing some
experimenting with some bits that we hope might start to serve as a
foundation for a new broker core for 6.0. Now that we're kind of out of
the investigation phase it seems like a good time to start some
discussion on what we've been looking at. The primary goals that we had
in mind were:

-Improved Scalability (particularly in threading model)
-Improved Performance.
-Improved Memory Management / Flow Control model
-Improved/Simplified Queue and Cursored Queue.
-Flexible protocol handler support. 
-Better modularization of components. 

So with that in mind, over the next couple of days/weeks I though we
could get a couple of threads going around the above points. First off
to get started looking at some of the stuff:

It is located in the activemq sandbox at:
https://svn.apache.org/repos/asf/activemq/sandbox/activemq-flow 

Architecture Notes
For a detailed list of architecture notes, check out the webgen
directory, the README file there gives instructions on how to generate
architecture notes. This generated site also contains a listing of modules
and dependencies.

Running some test cases:

There are few test cases that should help give you a feel for some of
this stuff.

When running these test cases you should be sure to use the -server jvm
argument:

1.	activemq-queue module org.apache.activemq.flow.MockBrokerTest.
This is a good starting point for looking at some of the flow control,
dispatcher and queue modules. It is a very basic MockBroker which uses a
lightweight proto-buf based protocol. You should feel free to play
around with some of the hardcoded options in the test (e.g. ptp=true,
tcp=true, threadsPerDispatcher etc).
2.	activemq-openwire module
org.apache.activemq.broker.SharedQueuePerfTest. This test does
performance testing against the SharedQueue implementation and is a good
place to start looking at queue persistence and the CursoredQueue. It
employs openwire messages with using network connecitons. Check out
org.apache.activemq.broker.BrokerQueueStore which currently hardcodes
the queue sizes; you can change DEFAULT_SHARED_QUEUE_PAGING_THRESHOLD
and the associated persistence policy to play around with paging.
3.	activemq-all module
org.apache.activemq.broker.openwire.OpenWireBrokerTest. This test uses
the full blown broker, adds in a tcp transport layert and provides
options for persistence and durable subscriptions (albeit asynchronous
at the moment). It extends org.apache.activemq.broker.BrokerTest in the
activemq-broker module which has some additional hard coded options to
play around with. Again refer to
org.apache.activemq.broker.BrokerQueueStore to mess around with queue
sizes.

It is still a work in progress, so any input would be a tremendous help!

Cheers,
Colin