Re: EIP - Message Broker Pattern

[Ashwin Karpe <as...@progress.com>]

Hi,

Yes, a Message Broker has several responsibilities that can get difficult
and unwieldy to manage as demands on the broker grow.

The Message Broker not only has to manage channel lifecycles, message
routing, message persistence in case App2 does not consume messages at the
rate App1 sends messages etc. These demands if not carefully measured for
performance degradation can cause a message broker to reach a point of
diminishing returns. It is more effective to split the broker at his point
into a network of cooperative brokers that take bring the problem back under
control by improving scale and delegating processing, management and client
support responsibilities to other brokers in the network.

The interesting fact is that any communication paradigm that has need for
asynchronous communication paired with service level guarantees (reiability
etc) need a mediating element aka broker to deliver this capability :-).
Ain't no way around it...

Cheers,

Ashwin...




tnabil wrote:
> 
> Hi,
> 
> As agreed, I'm prefixing the title of this post with "EIP" to denote that
> it's a general post on Enterprise Integration Patterns (not Camel
> specific).
> 
> Message Broker is one of the patterns described in the EIP book. I believe
> that this pattern is also the basis of ESB products currently being
> marketed by most vendors. From the EIP site, I quote:
> 
> 
> 
>> Use a central Message Broker that can receive messages from multiple
>> destinations, determine the correct destination and route the message to
>> the correct channel. Implement the internals of the Message Broker using
>> the design patterns presented in this chapter.
>> 
> 
> In the book, the author admits that the usage of this pattern may result
> in a bloat in the amount of functionality implemented inside the broker
> and suggests a federated approach to solve this problem.
> 
> Nevertheless, I found the treatment of this federated approach to be very
> thin.
> 
> For example, if an application App1 is connected to broker B1 and needs to
> send a message of type M2 which is handled by an application App2
> connected to broker B2, then how are the different channels configured to
> support this?
> 
> Does there need to be a channel on B1 which will accept messages of type
> M2 and then B1 would route it to B2 which would in turn route it to App2?
> 
> Wouldn't that mean that each broker would need to have channels for all
> messages which are sent by applications connected to it to other
> applications connected to other borkers? Wouldn't that have an impact on
> the number of channels, the amount of routing configuration and the
> managability of the solution?
> 
> Or is it that a borker would have a single channel which is not a Datatype
> Channel and all applications connected to it would send all messages to
> this channel and then the broker would determine whether they're to be
> routed to an application connected to it or through another broker?
> Wouldn't that make the required routing conifguration for this channel
> quite complex?
> 
> Isn't that why most people find ESB implementations to be bloated?
> 
> Wouldn't the broker in general become a single point of failure for all
> applications connected to it?
> 
> Is there a better approach for implementing this pattern? Has anyone
> covered this topic in more detail?
> 
> I'd appreciate your thoughts on the subject.
> 
> Thanks,
> Tarek Nabil
> 


-----
--- 
Ashwin Karpe, Principal Consultant, PS - Opensource Center of Competence 
Progress Software Corporation
14 Oak Park Drive
Bedford, MA 01730
--- 
+1-972-304-9084 (Office) 
+1-972-971-1700 (Mobile) 
---- 
Blog: http://opensourceknowledge.blogspot.com/


-- 
View this message in context: http://www.nabble.com/EIP---Message-Broker-Pattern-tp25676225p25706980.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: EIP - Message Broker Pattern

[Charles Moulliard <cm...@gmail.com>]

You are right David. Implementing a scalable and reliable platform is
certainly the most difficult thing to do and consume a lot of time /
resources between teams involved in infrastructure management.
Personnaly, I think that the scenario must be analyzed regarding to
the project or if there is a decision about the company to work with
ActiveMq as the global queueing engine, than you have to consider the
approach described here
(http://activemq.apache.org/networks-of-brokers.html) very carefully.

Other solutions could be investigated to support scalability in an
environment like :

- NMR clustering : http://servicemix.apache.org/SMX4NMR/13-clustering.html
- Loadbalance requests (through a HTTP proxy) to camel
endpoints/routes using RESTFull services
- ...

Regards,

Charles Moulliard
Senior Enterprise Architect
Apache Camel Committer

*****************************
blog : http://cmoulliard.blogspot.com
twitter : http://twitter.com/cmoulliard



On Wed, Oct 7, 2009 at 8:43 AM, tnabil <ta...@gmail.com> wrote:
>
> I went through some of the content and what I came to understand was that in
> a "store and forward" topology, manual replication of queues and topics
> would be required; please correct me if I'm wrong.
>
> It also seems that a lot of configuration and fine tuning would be required.
>
> I'm not sure if more expensive products like IBM Message Broker would
> provide visual tools that would make this task easier.
>
> What I feel, though, is that the only reason one would resort to such an
> approach would be (as also mentioned in the documentation) that the borkers
> need to be distributed across a WAN. Otherwise, the overhead of configuring
> the store and forward network seems to be a lot worse than having everything
> in one broker. For performance, of course, clustering can be done.
>
> Your thoughts?
>
>
> David Roussel wrote:
>>
>> Some of the issues are discussed here:
>> http://activemq.apache.org/networks-of-brokers.html
>>
>
>
> --
> View this message in context: http://www.nabble.com/EIP---Message-Broker-Pattern-tp25676225p25781296.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>

Re: EIP - Message Broker Pattern

[tnabil <ta...@gmail.com>]

I went through some of the content and what I came to understand was that in
a "store and forward" topology, manual replication of queues and topics
would be required; please correct me if I'm wrong.

It also seems that a lot of configuration and fine tuning would be required.

I'm not sure if more expensive products like IBM Message Broker would
provide visual tools that would make this task easier.

What I feel, though, is that the only reason one would resort to such an
approach would be (as also mentioned in the documentation) that the borkers
need to be distributed across a WAN. Otherwise, the overhead of configuring
the store and forward network seems to be a lot worse than having everything
in one broker. For performance, of course, clustering can be done.

Your thoughts? 


David Roussel wrote:
> 
> Some of the issues are discussed here:
> http://activemq.apache.org/networks-of-brokers.html
> 


-- 
View this message in context: http://www.nabble.com/EIP---Message-Broker-Pattern-tp25676225p25781296.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: EIP - Message Broker Pattern

[David Roussel <na...@diroussel.xsmail.com>]

Some of the issues are discussed here:
http://activemq.apache.org/networks-of-brokers.html


tnabil wrote:
> 
> Thanks, Ashwin, for your response.
> 
> It's exactly the federated approach (what you referred to as a network of
> brokers) that I'm interested in.
> 
> In particular, how are the channels (or virtualized services for all that
> matters) structured? Are they replicated across all the brokers in the
> network?
> 
> If yes, should the different brokers manage the routing among them
> automatically? Do the current products in the market support that?
> 
> 

-- 
View this message in context: http://www.nabble.com/EIP---Message-Broker-Pattern-tp25676225p25767802.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: EIP - Message Broker Pattern

[tnabil <ta...@gmail.com>]

Thanks, Ashwin, for your response.

It's exactly the federated approach (what you referred to as a network of
brokers) that I'm interested in.

In particular, how are the channels (or virtualized services for all that
matters) structured? Are they replicated across all the brokers in the
network?

If yes, should the different brokers manage the routing among them
automatically? Do the current products in the market support that?


Ashwin Karpe wrote:
> 
> Hi,
> 
> Yes, a Message Broker has several responsibilities that can get difficult
> and unwieldy to manage as demands on the broker grow.
> 
> The Message Broker not only has to manage channel lifecycles, message
> routing, message persistence in case App2 does not consume messages at the
> rate App1 sends messages etc. These demands if not carefully measured for
> performance degradation can cause a message broker to reach a point of
> diminishing returns. It is more effective to split the broker at his point
> into a network of cooperative brokers that take bring the problem back
> under control by improving scale and delegating processing, management and
> client support responsibilities to other brokers in the network.
> 
> The interesting fact is that any communication paradigm that has need for
> asynchronous transport paired with service level guarantees (reliability,
> message expiry etc) need a mediating element aka broker to transparently
> deliver this capability :-). 
> 
> Ain't no way around it... Any attempt to change this simply forces
> synchrony and complexity into the transport layer. %-|
> 
> Cheers,
> 
> Ashwin...
> 
> 
> 
> 
> tnabil wrote:
>> 
>> Hi,
>> 
>> As agreed, I'm prefixing the title of this post with "EIP" to denote that
>> it's a general post on Enterprise Integration Patterns (not Camel
>> specific).
>> 
>> Message Broker is one of the patterns described in the EIP book. I
>> believe that this pattern is also the basis of ESB products currently
>> being marketed by most vendors. From the EIP site, I quote:
>> 
>> 
>> 
>>> Use a central Message Broker that can receive messages from multiple
>>> destinations, determine the correct destination and route the message to
>>> the correct channel. Implement the internals of the Message Broker using
>>> the design patterns presented in this chapter.
>>> 
>> 
>> In the book, the author admits that the usage of this pattern may result
>> in a bloat in the amount of functionality implemented inside the broker
>> and suggests a federated approach to solve this problem.
>> 
>> Nevertheless, I found the treatment of this federated approach to be very
>> thin.
>> 
>> For example, if an application App1 is connected to broker B1 and needs
>> to send a message of type M2 which is handled by an application App2
>> connected to broker B2, then how are the different channels configured to
>> support this?
>> 
>> Does there need to be a channel on B1 which will accept messages of type
>> M2 and then B1 would route it to B2 which would in turn route it to App2?
>> 
>> Wouldn't that mean that each broker would need to have channels for all
>> messages which are sent by applications connected to it to other
>> applications connected to other borkers? Wouldn't that have an impact on
>> the number of channels, the amount of routing configuration and the
>> managability of the solution?
>> 
>> Or is it that a borker would have a single channel which is not a
>> Datatype Channel and all applications connected to it would send all
>> messages to this channel and then the broker would determine whether
>> they're to be routed to an application connected to it or through another
>> broker? Wouldn't that make the required routing conifguration for this
>> channel quite complex?
>> 
>> Isn't that why most people find ESB implementations to be bloated?
>> 
>> Wouldn't the broker in general become a single point of failure for all
>> applications connected to it?
>> 
>> Is there a better approach for implementing this pattern? Has anyone
>> covered this topic in more detail?
>> 
>> I'd appreciate your thoughts on the subject.
>> 
>> Thanks,
>> Tarek Nabil
>> 
> 
> 

-- 
View this message in context: http://www.nabble.com/EIP---Message-Broker-Pattern-tp25676225p25751260.html
Sent from the Camel - Users mailing list archive at Nabble.com.