You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by Hiram Chirino <hi...@hiramchirino.com> on 2009/02/11 21:34:37 UTC
New sandbox activemq-flow module
Hi,
Just wanted to let you know about a new activemq-flow little module
that I added to the ActiveMQ sandbox. It's a little experiment in
trying to do more efficient flow control and thread handling in todays
multi-core world. Its checked in at:
https://svn.apache.org/repos/asf/activemq/sandbox/activemq-flow
The 2 major bits of it are basically:
1) an event based thread/dispatching model which tries to avoid
cross thread contention
2) A flow control model where dispatching occurs across flow
controlled sinks and sources.
The flow control bits allow us to more easily implement things that
are missing in the current ActiveMQ like priority messaging and to do
better broker memory sizing based on connections to the broker.
Up to this point this has mostly been a collaborative experiment
between Colin Macnaughton (my highly esteemed coworker) and myself,
but I think it's a good time open this out to the rest of the
community to see if anyone would be interested in helping out in this
little experiment.
Things that still need to get done are:
- clean up / consolidate some of the unused bits. For example
SingleFlowPriorityQueue should get replaced with the
ExclusivePriorityQueue
- get the clustering working with the priority queues
- Update the mock test cases so it does real IO
I think once we get there, then we can start to do better apple to
apples comparisons to the existing broker architecture in terms of
performance, memory usage, and cpu usage.
--
Regards,
Hiram
Blog: http://hiramchirino.com
Open Source SOA
http://fusesource.com/
Re: New sandbox activemq-flow module
Posted by Hiram Chirino <ch...@gmail.com>.
FYI.. I just udpated the module to use the ActiveMQ transports so that
way it's testing against real IO.
The wire protocol is just Object Serialization based so it's not very
optimized but at least it should help
us see how the thread models interact.
On Thu, Feb 12, 2009 at 12:24 PM, Colin MacNaughton
<co...@gmail.com> wrote:
>
> Hi Everyone,
>
> As this is my first post to the dev list, I thought I might first share a
> little bit about my background. I started in messaging nine years ago at
> Progress Software working with SonicMQ. With the recent merger between IONA
> and Progress Software, I've happily been given the opportunity to begin
> working with ActiveMQ. I'm very much looking forward to sharing my messaging
> experience from SonicMQ and learning more from the impressive community
> around ActiveMQ.
>
> Anyway as Hiram mentioned I've been working with him on the activemq-flow
> module. One of the things that I find most interesting about this experiment
> is that it tries to boil messaging down into the management and control of
> message flows between sources and sinks. When thought of this way a message
> broker or network of brokers can be represented in a directed flow graph
> with queues as the vertices and message flows as the directed edges. Of
> course, we all know there is a lot more to messaging than this, but to me
> the concept of a flow graph presents some convenient advantages in terms of
> visibility (metrics and connectivity relationships, bottleneck detection) as
> well as management (policies, prioritization etc) to the users and
> administrators of a messaging system.
>
> I'm currently working on the dispatching component of this that Hiram
> mentioned in which we are trying to reduce threading contention between
> queues (sources and sinks). The theory is that by using the flow graph we
> can partition unrelated subcomponents of the graph onto different threads,
> thereby reducing thread contention between unrelated resources, and load
> balance the work amongst a small number of threads using metrics culled from
> the graph. The graphing/metrics component is not really there at the moment
> because the graphing library we were using wasn't really Apache friendly, so
> if anyone knows of a good graphing library please let me know!
>
> Looking forward to working with all of you, Colin
>
> colin.macnaughton@gmail.com
>
>
>
> Hiram Chirino wrote:
>>
>> Hi,
>>
>> Just wanted to let you know about a new activemq-flow little module
>> that I added to the ActiveMQ sandbox. It's a little experiment in
>> trying to do more efficient flow control and thread handling in todays
>> multi-core world. Its checked in at:
>> https://svn.apache.org/repos/asf/activemq/sandbox/activemq-flow
>>
>> The 2 major bits of it are basically:
>> 1) an event based thread/dispatching model which tries to avoid
>> cross thread contention
>> 2) A flow control model where dispatching occurs across flow
>> controlled sinks and sources.
>>
>> The flow control bits allow us to more easily implement things that
>> are missing in the current ActiveMQ like priority messaging and to do
>> better broker memory sizing based on connections to the broker.
>>
>> Up to this point this has mostly been a collaborative experiment
>> between Colin Macnaughton (my highly esteemed coworker) and myself,
>> but I think it's a good time open this out to the rest of the
>> community to see if anyone would be interested in helping out in this
>> little experiment.
>>
>> Things that still need to get done are:
>> - clean up / consolidate some of the unused bits. For example
>> SingleFlowPriorityQueue should get replaced with the
>> ExclusivePriorityQueue
>> - get the clustering working with the priority queues
>> - Update the mock test cases so it does real IO
>>
>> I think once we get there, then we can start to do better apple to
>> apples comparisons to the existing broker architecture in terms of
>> performance, memory usage, and cpu usage.
>>
>> --
>> Regards,
>> Hiram
>>
>> Blog: http://hiramchirino.com
>>
>> Open Source SOA
>> http://fusesource.com/
>>
>>
>
> --
> View this message in context: http://www.nabble.com/New-sandbox-activemq-flow-module-tp21963659p21980708.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>
>
--
Regards,
Hiram
Blog: http://hiramchirino.com
Open Source SOA
http://fusesource.com/
Re: New sandbox activemq-flow module
Posted by Colin MacNaughton <co...@gmail.com>.
Hi Everyone,
As this is my first post to the dev list, I thought I might first share a
little bit about my background. I started in messaging nine years ago at
Progress Software working with SonicMQ. With the recent merger between IONA
and Progress Software, I've happily been given the opportunity to begin
working with ActiveMQ. I'm very much looking forward to sharing my messaging
experience from SonicMQ and learning more from the impressive community
around ActiveMQ.
Anyway as Hiram mentioned I've been working with him on the activemq-flow
module. One of the things that I find most interesting about this experiment
is that it tries to boil messaging down into the management and control of
message flows between sources and sinks. When thought of this way a message
broker or network of brokers can be represented in a directed flow graph
with queues as the vertices and message flows as the directed edges. Of
course, we all know there is a lot more to messaging than this, but to me
the concept of a flow graph presents some convenient advantages in terms of
visibility (metrics and connectivity relationships, bottleneck detection) as
well as management (policies, prioritization etc) to the users and
administrators of a messaging system.
I'm currently working on the dispatching component of this that Hiram
mentioned in which we are trying to reduce threading contention between
queues (sources and sinks). The theory is that by using the flow graph we
can partition unrelated subcomponents of the graph onto different threads,
thereby reducing thread contention between unrelated resources, and load
balance the work amongst a small number of threads using metrics culled from
the graph. The graphing/metrics component is not really there at the moment
because the graphing library we were using wasn't really Apache friendly, so
if anyone knows of a good graphing library please let me know!
Looking forward to working with all of you, Colin
colin.macnaughton@gmail.com
Hiram Chirino wrote:
>
> Hi,
>
> Just wanted to let you know about a new activemq-flow little module
> that I added to the ActiveMQ sandbox. It's a little experiment in
> trying to do more efficient flow control and thread handling in todays
> multi-core world. Its checked in at:
> https://svn.apache.org/repos/asf/activemq/sandbox/activemq-flow
>
> The 2 major bits of it are basically:
> 1) an event based thread/dispatching model which tries to avoid
> cross thread contention
> 2) A flow control model where dispatching occurs across flow
> controlled sinks and sources.
>
> The flow control bits allow us to more easily implement things that
> are missing in the current ActiveMQ like priority messaging and to do
> better broker memory sizing based on connections to the broker.
>
> Up to this point this has mostly been a collaborative experiment
> between Colin Macnaughton (my highly esteemed coworker) and myself,
> but I think it's a good time open this out to the rest of the
> community to see if anyone would be interested in helping out in this
> little experiment.
>
> Things that still need to get done are:
> - clean up / consolidate some of the unused bits. For example
> SingleFlowPriorityQueue should get replaced with the
> ExclusivePriorityQueue
> - get the clustering working with the priority queues
> - Update the mock test cases so it does real IO
>
> I think once we get there, then we can start to do better apple to
> apples comparisons to the existing broker architecture in terms of
> performance, memory usage, and cpu usage.
>
> --
> Regards,
> Hiram
>
> Blog: http://hiramchirino.com
>
> Open Source SOA
> http://fusesource.com/
>
>
--
View this message in context: http://www.nabble.com/New-sandbox-activemq-flow-module-tp21963659p21980708.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
Re: New sandbox activemq-flow module
Posted by Hiram Chirino <ch...@gmail.com>.
Thanks! Fixed.
On Fri, Feb 13, 2009 at 3:41 PM, Bruce Snyder <br...@gmail.com> wrote:
> On Wed, Feb 11, 2009 at 1:34 PM, Hiram Chirino <hi...@hiramchirino.com> wrote:
>> Hi,
>>
>> Just wanted to let you know about a new activemq-flow little module
>> that I added to the ActiveMQ sandbox. It's a little experiment in
>> trying to do more efficient flow control and thread handling in todays
>> multi-core world. Its checked in at:
>> https://svn.apache.org/repos/asf/activemq/sandbox/activemq-flow
>>
>> The 2 major bits of it are basically:
>> 1) an event based thread/dispatching model which tries to avoid
>> cross thread contention
>> 2) A flow control model where dispatching occurs across flow
>> controlled sinks and sources.
>>
>> The flow control bits allow us to more easily implement things that
>> are missing in the current ActiveMQ like priority messaging and to do
>> better broker memory sizing based on connections to the broker.
>>
>> Up to this point this has mostly been a collaborative experiment
>> between Colin Macnaughton (my highly esteemed coworker) and myself,
>> but I think it's a good time open this out to the rest of the
>> community to see if anyone would be interested in helping out in this
>> little experiment.
>>
>> Things that still need to get done are:
>> - clean up / consolidate some of the unused bits. For example
>> SingleFlowPriorityQueue should get replaced with the
>> ExclusivePriorityQueue
>> - get the clustering working with the priority queues
>> - Update the mock test cases so it does real IO
>>
>> I think once we get there, then we can start to do better apple to
>> apples comparisons to the existing broker architecture in terms of
>> performance, memory usage, and cpu usage.
>
> Just a couple of things that I noticed at a glance. There is
> compilation dependency in the POM for an artifact named Colt that
> doesn't appear to be used in any classes. Furthermore, Colt has no
> explicit license and some of the packages Colt includes use the LGPL
> license (http://acs.lbl.gov/~hoschek/colt/license.html).
>
> Also, the com/progress directories in src/main/java and src/test/java
> need to be removed.
>
> Bruce
> --
> perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> Apache ActiveMQ - http://activemq.org/
> Apache Camel - http://activemq.org/camel/
> Apache ServiceMix - http://servicemix.org/
>
> Blog: http://bruceblog.org/
>
--
Regards,
Hiram
Blog: http://hiramchirino.com
Open Source SOA
http://fusesource.com/
RE: New sandbox activemq-flow module
Posted by Colin MacNaughton <co...@gmail.com>.
Hi Bruce,
Thanks for checking it out and the pointers! I'll get those cleaned up.
Colin
-----Original Message-----
From: Bruce Snyder [mailto:bruce.snyder@gmail.com]
Sent: Friday, February 13, 2009 12:42 PM
To: dev@activemq.apache.org
Subject: Re: New sandbox activemq-flow module
On Wed, Feb 11, 2009 at 1:34 PM, Hiram Chirino <hi...@hiramchirino.com>
wrote:
> Hi,
>
> Just wanted to let you know about a new activemq-flow little module
> that I added to the ActiveMQ sandbox. It's a little experiment in
> trying to do more efficient flow control and thread handling in todays
> multi-core world. Its checked in at:
> https://svn.apache.org/repos/asf/activemq/sandbox/activemq-flow
>
> The 2 major bits of it are basically:
> 1) an event based thread/dispatching model which tries to avoid
> cross thread contention
> 2) A flow control model where dispatching occurs across flow
> controlled sinks and sources.
>
> The flow control bits allow us to more easily implement things that
> are missing in the current ActiveMQ like priority messaging and to do
> better broker memory sizing based on connections to the broker.
>
> Up to this point this has mostly been a collaborative experiment
> between Colin Macnaughton (my highly esteemed coworker) and myself,
> but I think it's a good time open this out to the rest of the
> community to see if anyone would be interested in helping out in this
> little experiment.
>
> Things that still need to get done are:
> - clean up / consolidate some of the unused bits. For example
> SingleFlowPriorityQueue should get replaced with the
> ExclusivePriorityQueue
> - get the clustering working with the priority queues
> - Update the mock test cases so it does real IO
>
> I think once we get there, then we can start to do better apple to
> apples comparisons to the existing broker architecture in terms of
> performance, memory usage, and cpu usage.
Just a couple of things that I noticed at a glance. There is
compilation dependency in the POM for an artifact named Colt that
doesn't appear to be used in any classes. Furthermore, Colt has no
explicit license and some of the packages Colt includes use the LGPL
license (http://acs.lbl.gov/~hoschek/colt/license.html).
Also, the com/progress directories in src/main/java and src/test/java
need to be removed.
Bruce
--
perl -e 'print
unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/
Blog: http://bruceblog.org/
Re: New sandbox activemq-flow module
Posted by Bruce Snyder <br...@gmail.com>.
On Wed, Feb 11, 2009 at 1:34 PM, Hiram Chirino <hi...@hiramchirino.com> wrote:
> Hi,
>
> Just wanted to let you know about a new activemq-flow little module
> that I added to the ActiveMQ sandbox. It's a little experiment in
> trying to do more efficient flow control and thread handling in todays
> multi-core world. Its checked in at:
> https://svn.apache.org/repos/asf/activemq/sandbox/activemq-flow
>
> The 2 major bits of it are basically:
> 1) an event based thread/dispatching model which tries to avoid
> cross thread contention
> 2) A flow control model where dispatching occurs across flow
> controlled sinks and sources.
>
> The flow control bits allow us to more easily implement things that
> are missing in the current ActiveMQ like priority messaging and to do
> better broker memory sizing based on connections to the broker.
>
> Up to this point this has mostly been a collaborative experiment
> between Colin Macnaughton (my highly esteemed coworker) and myself,
> but I think it's a good time open this out to the rest of the
> community to see if anyone would be interested in helping out in this
> little experiment.
>
> Things that still need to get done are:
> - clean up / consolidate some of the unused bits. For example
> SingleFlowPriorityQueue should get replaced with the
> ExclusivePriorityQueue
> - get the clustering working with the priority queues
> - Update the mock test cases so it does real IO
>
> I think once we get there, then we can start to do better apple to
> apples comparisons to the existing broker architecture in terms of
> performance, memory usage, and cpu usage.
Just a couple of things that I noticed at a glance. There is
compilation dependency in the POM for an artifact named Colt that
doesn't appear to be used in any classes. Furthermore, Colt has no
explicit license and some of the packages Colt includes use the LGPL
license (http://acs.lbl.gov/~hoschek/colt/license.html).
Also, the com/progress directories in src/main/java and src/test/java
need to be removed.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/
Blog: http://bruceblog.org/