You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Rob Davies <ra...@gmail.com> on 2006/02/19 10:49:17 UTC
Re: [activemq-user] Network of Brokers ...
Networks are being updated currently for release 4.0
Consumers are propagated around the network - but within constraints:
1. networkTTL - number of hops (brokers) that this information will
pass through - networkTTL also applies to messages dispatched as well
2. exclusiveDestinations - can exclude destinations from network
traffic (can use wild-cards etc)
3. inclusiveDestinations - can specify only destinations which are in
the inclusive list to be propagated around the network
I'll update the wiki with this information once we're satisfied its
working.
cheers,
Rob
On 13 Feb 2006, at 23:15, Sridhar Komandur wrote:
> Hi,
>
> 1. Can anyone clarify how the routing info is shared within Network of
> Brokers
> - is the producer/subscription info known to all the brokers
> in the
> network ? As I understand,
> 4.0 follows demand-forwarding for the actual message flow.
>
> 2. Has anyone /deployed/thought of/ an architecture where multiple
> "Network
> of Brokers" clusters would be deployed ?
> (not unlike ISP domains, within the same enterprise - for
> capacity or
> ownership reasons).
> I was wondering how "routes" can be selectively disseminated
> between
> the clusters.
>
>
> Thanks for any insights into the above.
>
> Thanks
> Regards
> - Sridhar Komandur
Re: [activemq-user] Network of Brokers ...
Posted by Rob Davies <ra...@gmail.com>.
Hi John,
yes it should! well spotted :)
cheers,
Rob
On 27 Feb 2006, at 19:00, John Heitmann wrote:
>
> On Feb 27, 2006, at 12:44 AM, Rob Davies wrote:
>> ActiveMQ tags messages with the brokers the message has passed
>> through - so messages can be filtered out before they are sent to
>> a broker that message has already visited
>
> Should the first if clause of
> org.apache.activemq.network.DemandForwardingBridge.matchesForwardingFi
> lter() have an explicit 'return false;'? It looks like it logs the
> loop but doesn't filter it (I haven't watched this path in action
> though, it might do the right thing in practice).
>
> John
Re: [activemq-user] Network of Brokers ...
Posted by John Heitmann <jh...@gmail.com>.
On Feb 27, 2006, at 12:44 AM, Rob Davies wrote:
> ActiveMQ tags messages with the brokers the message has passed
> through - so messages can be filtered out before they are sent to a
> broker that message has already visited
Should the first if clause of
org.apache.activemq.network.DemandForwardingBridge.matchesForwardingFilt
er() have an explicit 'return false;'? It looks like it logs the loop
but doesn't filter it (I haven't watched this path in action though,
it might do the right thing in practice).
John
Re: [activemq-user] Network of Brokers ...
Posted by Rob Davies <ra...@gmail.com>.
Hi Srdhar,
answers inlined:
On 24 Feb 2006, at 19:41, Sridhar Komandur wrote:
> Thanks Rob ... I have gone through the link, andcould you please
> clarify :
>
> 1. when multiple routes exist btw producer and consumer brokers,
> are you suggesting
> that under 'normal' circumstances duplicates are prevented ? What
> circumstances
> is it prevented (and which ones result in duplicates ?).
ActiveMQ tags messages with the brokers the message has passed
through - so messages can be filtered out before they are sent to a
broker that message has already visited -
i.e. A -> B -> C -> A (a ring) -
A message originating at broker A - would not be passed from broker C
to broker A (so loops like this are avoided).
However - (excuse the drawing)
B
/ \
A D
\ /
C
A message sent from A to both B and C - B and C both forward to D - D
would get a duplicate of the original.
Now it's possible to filter out duplicates on broker D - by saving a
history of message ids that have passed through broker D - but then
there are a lot of edge cases, especially around re-delivery and
queues - which makes it difficult in practice to do.
>
> 2. NetworkConnector 'name' property:
> name bridge name of the network - for more than one network
> connector between the same two brokers - use different names
>
> example (
>
> Producer-- A- B-C-D ---Consumer
> | |
> +---E---+
>
> 2.1 Does 'name' apply in the above scenario - A and D have multiple
> paths, but not direct NetworkConnectors.
No - it only applies to scenarios like this:
->
A B
->
Note the direction (as networks are single plex) - a unique name for
the network would not apply to the usual case for networks:
A <-> B
>
> 2.2 How does A route based on 'name', using the above example
> (cyclic case) ?
The name is not used for routing - only for creating a unique but
repeatable client ID that is used by the network connector.
>
> 3. I would prefer not having to be restricted to acyclic (tree)
> graphs, for several reasons:
> - more flexibility in routing (e.g., admin reasons)
> - single point of failures (I undertand failover tranport will
> create NetworkConnector to another broker - but the /flash flood
> effects of the distributed system may lead to instability)
> - traffic saturation and inability to take advantage of multiple
> paths (load balancing)
> for different flows.
>
ok - but hopefully you can use a combination of master/slave and
acyclic networks to get the same solution.
> 4. I would like to run some preliminary ideas by you offline to
> deal with duplicate messages - unless this feature is deemed off
> the ActiveMQ roadmap.
>
If possible - can we keep the discussion on the list - I'm copying to
dev - that way we'll get the benefit of the community input as well :).
> Thank you Sir !
> Regards
> - Sri
Thank you!
cheers,
Rob
>
> On 2/24/06, Rob Davies <ra...@gmail.com> wrote:
> Hi Sridhar,
>
> ActiveMQ does try and prevent sending duplicates - but there are
> circumstances where it would be possible for multiple routes across a
> network to result in duplicates. I'd advise you to only use acyclic
> graphs, or where this can't be avoided, use well named destinations
> and network filters.
>
> See http://docs.codehaus.org/display/ACTIVEMQ/Networks+of+Brokers
>
> cheers,
>
> Rob
>
> On 23 Feb 2006, at 22:08, Sridhar Komandur wrote:
>
> > Rob, thanks for your email.
> >
> > Would you mind clarifying item 1 below, what if there are more than
> > one path
> > between producer and a consumer for the given networkTTL ?
> >
> > Will the consumer_broker receive as many messages as the number of
> > paths ?
> > (I hope not). Could you please elaborate on the details of message
> > routing
> > in this case ?
> >
> > Thanks!
> > Regards
> > - Sridhar
> >
> > On 2/19/06, Rob Davies <ra...@gmail.com> wrote:
> >>
> >>
> >> Networks are being updated currently for release 4.0
> >> Consumers are propagated around the network - but within
> constraints:
> >>
> >> 1. networkTTL - number of hops (brokers) that this information will
> >> pass through - networkTTL also applies to messages dispatched as
> well
> >> 2. exclusiveDestinations - can exclude destinations from network
> >> traffic (can use wild-cards etc)
> >> 3. inclusiveDestinations - can specify only destinations which
> are in
> >> the inclusive list to be propagated around the network
> >>
> >>
> >> I'll update the wiki with this information once we're satisfied its
> >> working.
> >>
> >> cheers,
> >>
> >> Rob
> >>
> >> On 13 Feb 2006, at 23:15, Sridhar Komandur wrote:
> >>
> >>> Hi,
> >>>
> >>> 1. Can anyone clarify how the routing info is shared within
> >>> Network of
> >>> Brokers
> >>> - is the producer/subscription info known to all the brokers
> >>> in the
> >>> network ? As I understand,
> >>> 4.0 follows demand-forwarding for the actual message flow.
> >>>
> >>> 2. Has anyone /deployed/thought of/ an architecture where multiple
> >>> "Network
> >>> of Brokers" clusters would be deployed ?
> >>> (not unlike ISP domains, within the same enterprise - for
> >>> capacity or
> >>> ownership reasons).
> >>> I was wondering how "routes" can be selectively disseminated
> >>> between
> >>> the clusters.
> >>>
> >>>
> >>> Thanks for any insights into the above.
> >>>
> >>> Thanks
> >>> Regards
> >>> - Sridhar Komandur
> >>
> >>
>
>
Re: [activemq-user] Network of Brokers ...
Posted by Rob Davies <ra...@gmail.com>.
Hi Srdhar,
answers inlined:
On 24 Feb 2006, at 19:41, Sridhar Komandur wrote:
> Thanks Rob ... I have gone through the link, andcould you please
> clarify :
>
> 1. when multiple routes exist btw producer and consumer brokers,
> are you suggesting
> that under 'normal' circumstances duplicates are prevented ? What
> circumstances
> is it prevented (and which ones result in duplicates ?).
ActiveMQ tags messages with the brokers the message has passed
through - so messages can be filtered out before they are sent to a
broker that message has already visited -
i.e. A -> B -> C -> A (a ring) -
A message originating at broker A - would not be passed from broker C
to broker A (so loops like this are avoided).
However - (excuse the drawing)
B
/ \
A D
\ /
C
A message sent from A to both B and C - B and C both forward to D - D
would get a duplicate of the original.
Now it's possible to filter out duplicates on broker D - by saving a
history of message ids that have passed through broker D - but then
there are a lot of edge cases, especially around re-delivery and
queues - which makes it difficult in practice to do.
>
> 2. NetworkConnector 'name' property:
> name bridge name of the network - for more than one network
> connector between the same two brokers - use different names
>
> example (
>
> Producer-- A- B-C-D ---Consumer
> | |
> +---E---+
>
> 2.1 Does 'name' apply in the above scenario - A and D have multiple
> paths, but not direct NetworkConnectors.
No - it only applies to scenarios like this:
->
A B
->
Note the direction (as networks are single plex) - a unique name for
the network would not apply to the usual case for networks:
A <-> B
>
> 2.2 How does A route based on 'name', using the above example
> (cyclic case) ?
The name is not used for routing - only for creating a unique but
repeatable client ID that is used by the network connector.
>
> 3. I would prefer not having to be restricted to acyclic (tree)
> graphs, for several reasons:
> - more flexibility in routing (e.g., admin reasons)
> - single point of failures (I undertand failover tranport will
> create NetworkConnector to another broker - but the /flash flood
> effects of the distributed system may lead to instability)
> - traffic saturation and inability to take advantage of multiple
> paths (load balancing)
> for different flows.
>
ok - but hopefully you can use a combination of master/slave and
acyclic networks to get the same solution.
> 4. I would like to run some preliminary ideas by you offline to
> deal with duplicate messages - unless this feature is deemed off
> the ActiveMQ roadmap.
>
If possible - can we keep the discussion on the list - I'm copying to
dev - that way we'll get the benefit of the community input as well :).
> Thank you Sir !
> Regards
> - Sri
Thank you!
cheers,
Rob
>
> On 2/24/06, Rob Davies <ra...@gmail.com> wrote:
> Hi Sridhar,
>
> ActiveMQ does try and prevent sending duplicates - but there are
> circumstances where it would be possible for multiple routes across a
> network to result in duplicates. I'd advise you to only use acyclic
> graphs, or where this can't be avoided, use well named destinations
> and network filters.
>
> See http://docs.codehaus.org/display/ACTIVEMQ/Networks+of+Brokers
>
> cheers,
>
> Rob
>
> On 23 Feb 2006, at 22:08, Sridhar Komandur wrote:
>
> > Rob, thanks for your email.
> >
> > Would you mind clarifying item 1 below, what if there are more than
> > one path
> > between producer and a consumer for the given networkTTL ?
> >
> > Will the consumer_broker receive as many messages as the number of
> > paths ?
> > (I hope not). Could you please elaborate on the details of message
> > routing
> > in this case ?
> >
> > Thanks!
> > Regards
> > - Sridhar
> >
> > On 2/19/06, Rob Davies <ra...@gmail.com> wrote:
> >>
> >>
> >> Networks are being updated currently for release 4.0
> >> Consumers are propagated around the network - but within
> constraints:
> >>
> >> 1. networkTTL - number of hops (brokers) that this information will
> >> pass through - networkTTL also applies to messages dispatched as
> well
> >> 2. exclusiveDestinations - can exclude destinations from network
> >> traffic (can use wild-cards etc)
> >> 3. inclusiveDestinations - can specify only destinations which
> are in
> >> the inclusive list to be propagated around the network
> >>
> >>
> >> I'll update the wiki with this information once we're satisfied its
> >> working.
> >>
> >> cheers,
> >>
> >> Rob
> >>
> >> On 13 Feb 2006, at 23:15, Sridhar Komandur wrote:
> >>
> >>> Hi,
> >>>
> >>> 1. Can anyone clarify how the routing info is shared within
> >>> Network of
> >>> Brokers
> >>> - is the producer/subscription info known to all the brokers
> >>> in the
> >>> network ? As I understand,
> >>> 4.0 follows demand-forwarding for the actual message flow.
> >>>
> >>> 2. Has anyone /deployed/thought of/ an architecture where multiple
> >>> "Network
> >>> of Brokers" clusters would be deployed ?
> >>> (not unlike ISP domains, within the same enterprise - for
> >>> capacity or
> >>> ownership reasons).
> >>> I was wondering how "routes" can be selectively disseminated
> >>> between
> >>> the clusters.
> >>>
> >>>
> >>> Thanks for any insights into the above.
> >>>
> >>> Thanks
> >>> Regards
> >>> - Sridhar Komandur
> >>
> >>
>
>
Re: [activemq-user] Network of Brokers ...
Posted by Sridhar Komandur <an...@komandur.com>.
Thanks Rob ... I have gone through the link, andcould you please clarify :
1. when multiple routes exist btw producer and consumer brokers, are you
suggesting
that under 'normal' circumstances duplicates are prevented ? What
circumstances
is it prevented (and which ones result in duplicates ?).
2. NetworkConnector 'name' property:
name bridge name of the network - for more than one network connector
between the same two brokers - use different names
example (
Producer-- A- B-C-D ---Consumer
| |
+---E---+
2.1 Does 'name' apply in the above scenario - A and D have multiple paths,
but not direct NetworkConnectors.
2.2 How does A route based on 'name', using the above example (cyclic case)
?
3. I would prefer not having to be restricted to acyclic (tree) graphs, for
several reasons:
- more flexibility in routing (e.g., admin reasons)
- single point of failures (I undertand failover tranport will create
NetworkConnector to another broker - but the /flash flood effects of the
distributed system may lead to instability)
- traffic saturation and inability to take advantage of multiple paths
(load balancing)
for different flows.
4. I would like to run some preliminary ideas by you offline to deal with
duplicate messages - unless this feature is deemed off the ActiveMQ
roadmap.
Thank you Sir !
Regards
- Sri
On 2/24/06, Rob Davies <ra...@gmail.com> wrote:
>
> Hi Sridhar,
>
> ActiveMQ does try and prevent sending duplicates - but there are
> circumstances where it would be possible for multiple routes across a
> network to result in duplicates. I'd advise you to only use acyclic
> graphs, or where this can't be avoided, use well named destinations
> and network filters.
>
> See http://docs.codehaus.org/display/ACTIVEMQ/Networks+of+Brokers
>
> cheers,
>
> Rob
>
> On 23 Feb 2006, at 22:08, Sridhar Komandur wrote:
>
> > Rob, thanks for your email.
> >
> > Would you mind clarifying item 1 below, what if there are more than
> > one path
> > between producer and a consumer for the given networkTTL ?
> >
> > Will the consumer_broker receive as many messages as the number of
> > paths ?
> > (I hope not). Could you please elaborate on the details of message
> > routing
> > in this case ?
> >
> > Thanks!
> > Regards
> > - Sridhar
> >
> > On 2/19/06, Rob Davies <ra...@gmail.com> wrote:
> >>
> >>
> >> Networks are being updated currently for release 4.0
> >> Consumers are propagated around the network - but within constraints:
> >>
> >> 1. networkTTL - number of hops (brokers) that this information will
> >> pass through - networkTTL also applies to messages dispatched as well
> >> 2. exclusiveDestinations - can exclude destinations from network
> >> traffic (can use wild-cards etc)
> >> 3. inclusiveDestinations - can specify only destinations which are in
> >> the inclusive list to be propagated around the network
> >>
> >>
> >> I'll update the wiki with this information once we're satisfied its
> >> working.
> >>
> >> cheers,
> >>
> >> Rob
> >>
> >> On 13 Feb 2006, at 23:15, Sridhar Komandur wrote:
> >>
> >>> Hi,
> >>>
> >>> 1. Can anyone clarify how the routing info is shared within
> >>> Network of
> >>> Brokers
> >>> - is the producer/subscription info known to all the brokers
> >>> in the
> >>> network ? As I understand,
> >>> 4.0 follows demand-forwarding for the actual message flow.
> >>>
> >>> 2. Has anyone /deployed/thought of/ an architecture where multiple
> >>> "Network
> >>> of Brokers" clusters would be deployed ?
> >>> (not unlike ISP domains, within the same enterprise - for
> >>> capacity or
> >>> ownership reasons).
> >>> I was wondering how "routes" can be selectively disseminated
> >>> between
> >>> the clusters.
> >>>
> >>>
> >>> Thanks for any insights into the above.
> >>>
> >>> Thanks
> >>> Regards
> >>> - Sridhar Komandur
> >>
> >>
>
>
Re: [activemq-user] Network of Brokers ...
Posted by Rob Davies <ra...@gmail.com>.
Hi Sridhar,
ActiveMQ does try and prevent sending duplicates - but there are
circumstances where it would be possible for multiple routes across a
network to result in duplicates. I'd advise you to only use acyclic
graphs, or where this can't be avoided, use well named destinations
and network filters.
See http://docs.codehaus.org/display/ACTIVEMQ/Networks+of+Brokers
cheers,
Rob
On 23 Feb 2006, at 22:08, Sridhar Komandur wrote:
> Rob, thanks for your email.
>
> Would you mind clarifying item 1 below, what if there are more than
> one path
> between producer and a consumer for the given networkTTL ?
>
> Will the consumer_broker receive as many messages as the number of
> paths ?
> (I hope not). Could you please elaborate on the details of message
> routing
> in this case ?
>
> Thanks!
> Regards
> - Sridhar
>
> On 2/19/06, Rob Davies <ra...@gmail.com> wrote:
>>
>>
>> Networks are being updated currently for release 4.0
>> Consumers are propagated around the network - but within constraints:
>>
>> 1. networkTTL - number of hops (brokers) that this information will
>> pass through - networkTTL also applies to messages dispatched as well
>> 2. exclusiveDestinations - can exclude destinations from network
>> traffic (can use wild-cards etc)
>> 3. inclusiveDestinations - can specify only destinations which are in
>> the inclusive list to be propagated around the network
>>
>>
>> I'll update the wiki with this information once we're satisfied its
>> working.
>>
>> cheers,
>>
>> Rob
>>
>> On 13 Feb 2006, at 23:15, Sridhar Komandur wrote:
>>
>>> Hi,
>>>
>>> 1. Can anyone clarify how the routing info is shared within
>>> Network of
>>> Brokers
>>> - is the producer/subscription info known to all the brokers
>>> in the
>>> network ? As I understand,
>>> 4.0 follows demand-forwarding for the actual message flow.
>>>
>>> 2. Has anyone /deployed/thought of/ an architecture where multiple
>>> "Network
>>> of Brokers" clusters would be deployed ?
>>> (not unlike ISP domains, within the same enterprise - for
>>> capacity or
>>> ownership reasons).
>>> I was wondering how "routes" can be selectively disseminated
>>> between
>>> the clusters.
>>>
>>>
>>> Thanks for any insights into the above.
>>>
>>> Thanks
>>> Regards
>>> - Sridhar Komandur
>>
>>
Re: [activemq-user] Network of Brokers ...
Posted by Sridhar Komandur <an...@komandur.com>.
Rob, thanks for your email.
Would you mind clarifying item 1 below, what if there are more than one path
between producer and a consumer for the given networkTTL ?
Will the consumer_broker receive as many messages as the number of paths ?
(I hope not). Could you please elaborate on the details of message routing
in this case ?
Thanks!
Regards
- Sridhar
On 2/19/06, Rob Davies <ra...@gmail.com> wrote:
>
>
> Networks are being updated currently for release 4.0
> Consumers are propagated around the network - but within constraints:
>
> 1. networkTTL - number of hops (brokers) that this information will
> pass through - networkTTL also applies to messages dispatched as well
> 2. exclusiveDestinations - can exclude destinations from network
> traffic (can use wild-cards etc)
> 3. inclusiveDestinations - can specify only destinations which are in
> the inclusive list to be propagated around the network
>
>
> I'll update the wiki with this information once we're satisfied its
> working.
>
> cheers,
>
> Rob
>
> On 13 Feb 2006, at 23:15, Sridhar Komandur wrote:
>
> > Hi,
> >
> > 1. Can anyone clarify how the routing info is shared within Network of
> > Brokers
> > - is the producer/subscription info known to all the brokers
> > in the
> > network ? As I understand,
> > 4.0 follows demand-forwarding for the actual message flow.
> >
> > 2. Has anyone /deployed/thought of/ an architecture where multiple
> > "Network
> > of Brokers" clusters would be deployed ?
> > (not unlike ISP domains, within the same enterprise - for
> > capacity or
> > ownership reasons).
> > I was wondering how "routes" can be selectively disseminated
> > between
> > the clusters.
> >
> >
> > Thanks for any insights into the above.
> >
> > Thanks
> > Regards
> > - Sridhar Komandur
>
>