You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by dunnlow <du...@yahoo.com> on 2013/10/23 19:52:05 UTC
Best way to consuming same queue from two brokers?
Using Camel 2.10.3
My question: What is the best way to consume msgs from multiple
brokers/queues and feed those messages through a single route?
Explanation:
I currently consume from an activemq broker. I have several different
routes that consume messages from different queues on that broker. Now, the
admins have updated to a network of brokers.
The producers of the messages are using failover (there are only two brokers
at the moment). Because those producers could place msgs on either broker,
I want to check both brokers (I had a past issue on a different project
where the producers failed over briefly and sent msgs to the backup that
were never consumed even though I was using a failover url. My solution at
the time was to always check/consume from both primary and backup - ignoring
failover - using a second activemq config to the secondary broker).
My problem is that now I have many routes and I want msgs from either the
primary or backup jms queue to propagate through those routes without having
to duplicate the route for each broker. For example, I DONT want to do this
for each route (my actual routes are more complex than these):
<route>
<from uri="activemq:queue:Queue1"/>
<bean ref="MyBean" method="evaluate"/>
<to uri="direct:processit"/>
</route>
<route>
<from uri="activemq2:queue:Queue1"/>
<bean ref="MyBean" method="evaluate"/>
<to uri="direct:processit"/>
</route>
I realize I could make the route above a direct route and then consume from
each broker and send to it, but this still seems ugly:
<route>
<from uri="direct_activemq_Queue1/>
<bean ref="MyBean" method="evaluate"/>
<to uri="direct:processit"/>
</route>
<route>
<from uri="activemq1:queue:Queue1"/>
<to uri="direct:direct_activemq_Queue1"/>
</route>
<route>
<from uri="activemq2:queue:Queue1"/>
<to uri="direct:direct_activemq_Queue1"/>
</route>
I've considered combining into a single in-memory queue, but I've never done
that before and would worry about robustness of the queue (I can't lose any
messages).
What is the smart way to do this?
Thanks very much for any insight!
-J
--
View this message in context: http://camel.465427.n5.nabble.com/Best-way-to-consuming-same-queue-from-two-brokers-tp5742100.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Best way to consuming same queue from two brokers?
Posted by James Carman <ja...@carmanconsulting.com>.
Remember that a NoB is not a "cluster", per se. There is no
replication between brokers. So, if one broker goes down while there
are messages sitting on it, then those messages are lost until the
broker comes back online. If you're looking for high availability,
then you're barking up the wrong tree. You need to look into a
Master/Slave setup.
On Wed, Oct 23, 2013 at 2:57 PM, dunnlow <du...@yahoo.com> wrote:
> Thanks James. Well, yes... sort of. If a producer sends to one broker, I
> can consume from either as expected...IF both brokers are ok. My concern
> (that I have from my relayed past experience) is that if I use the failover
> uri and there is a problem, that my consumer will miss some of the message
> produced. Maybe this isn't a legitimate concern (although, in my
> skepticism's defense, the admin who configured it expressed a lack in
> confidence about it's configuration). Thus, I think the safest way forward
> may be to point a consumer to each of the brokers. Considering more brokers
> may come online, what I guess there would be ideally it to funnel all of the
> brokers into a single endpoint from which I could process. Does that make
> sense or am I way overcomplicating it? Thanks again.
>
>
> James Carman wrote
>> The NoB should be routing your messages for you, shouldn't it?
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Best-way-to-consuming-same-queue-from-two-brokers-tp5742100p5742105.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Best way to consuming same queue from two brokers?
Posted by dunnlow <du...@yahoo.com>.
Thanks James. Well, yes... sort of. If a producer sends to one broker, I
can consume from either as expected...IF both brokers are ok. My concern
(that I have from my relayed past experience) is that if I use the failover
uri and there is a problem, that my consumer will miss some of the message
produced. Maybe this isn't a legitimate concern (although, in my
skepticism's defense, the admin who configured it expressed a lack in
confidence about it's configuration). Thus, I think the safest way forward
may be to point a consumer to each of the brokers. Considering more brokers
may come online, what I guess there would be ideally it to funnel all of the
brokers into a single endpoint from which I could process. Does that make
sense or am I way overcomplicating it? Thanks again.
James Carman wrote
> The NoB should be routing your messages for you, shouldn't it?
--
View this message in context: http://camel.465427.n5.nabble.com/Best-way-to-consuming-same-queue-from-two-brokers-tp5742100p5742105.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Best way to consuming same queue from two brokers?
Posted by James Carman <ja...@carmanconsulting.com>.
The NoB should be routing your messages for you, shouldn't it?
On Wed, Oct 23, 2013 at 1:52 PM, dunnlow <du...@yahoo.com> wrote:
> Using Camel 2.10.3
>
> My question: What is the best way to consume msgs from multiple
> brokers/queues and feed those messages through a single route?
>
> Explanation:
>
> I currently consume from an activemq broker. I have several different
> routes that consume messages from different queues on that broker. Now, the
> admins have updated to a network of brokers.
>
> The producers of the messages are using failover (there are only two brokers
> at the moment). Because those producers could place msgs on either broker,
> I want to check both brokers (I had a past issue on a different project
> where the producers failed over briefly and sent msgs to the backup that
> were never consumed even though I was using a failover url. My solution at
> the time was to always check/consume from both primary and backup - ignoring
> failover - using a second activemq config to the secondary broker).
>
> My problem is that now I have many routes and I want msgs from either the
> primary or backup jms queue to propagate through those routes without having
> to duplicate the route for each broker. For example, I DONT want to do this
> for each route (my actual routes are more complex than these):
>
> <route>
> <from uri="activemq:queue:Queue1"/>
> <bean ref="MyBean" method="evaluate"/>
> <to uri="direct:processit"/>
> </route>
>
> <route>
> <from uri="activemq2:queue:Queue1"/>
> <bean ref="MyBean" method="evaluate"/>
> <to uri="direct:processit"/>
> </route>
>
> I realize I could make the route above a direct route and then consume from
> each broker and send to it, but this still seems ugly:
>
> <route>
> <from uri="direct_activemq_Queue1/>
> <bean ref="MyBean" method="evaluate"/>
> <to uri="direct:processit"/>
> </route>
>
> <route>
> <from uri="activemq1:queue:Queue1"/>
> <to uri="direct:direct_activemq_Queue1"/>
> </route>
>
> <route>
> <from uri="activemq2:queue:Queue1"/>
> <to uri="direct:direct_activemq_Queue1"/>
> </route>
>
> I've considered combining into a single in-memory queue, but I've never done
> that before and would worry about robustness of the queue (I can't lose any
> messages).
>
> What is the smart way to do this?
>
> Thanks very much for any insight!
>
> -J
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Best-way-to-consuming-same-queue-from-two-brokers-tp5742100.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Best way to consuming same queue from two brokers?
Posted by Christian Posta <ch...@gmail.com>.
Check what James said about using Master/Slave to prevent messages being
stuck on one broker too long in the event one of the brokers in your
Active-Active NoB goes down.
Also make sure that messages can make it back to their originating broker
if there are no consumers with the ConditionalNetworkBridgeFilter:
http://activemq.apache.org/networks-of-brokers.html
If you're still skeptical about the routingYou can check both brokers like
this:
<route>
<from uri="activemq:queue:Queue1"/>
<from uri="activemq2:queue:Queue1"/>
<bean ref="MyBean" method="evaluate"/>
<to uri="direct:processit"/>
</route>
On Thu, Oct 24, 2013 at 10:21 AM, dunnlow <du...@yahoo.com> wrote:
> Thanks Reji, I understand. My fear with SEDA is that I have a pretty high
> msg flow at times and if my app goes down and there are messages on the
> SEDA
> queue (that is, pulled off of various jms brokers/queues and waiting on a
> SEDA queue) the msgs will be lost. But I wonder, can a transaction work
> across that? For example, in a transaction, pull the msgs off of three
> different activemq instances and place in SEDA queue, but only consume one
> of the msgs from SEDA and then shut down the system. Can the transaction
> be
> maintained so that the two unconsumed msgs roll back to their respective
> brokers/queues? I feel some more testing coming on... :) Any thought?
> Thanks again, -J
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Best-way-to-consuming-same-queue-from-two-brokers-tp5742100p5742134.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Best way to consuming same queue from two brokers?
Posted by dunnlow <du...@yahoo.com>.
Thanks Reji, I understand. My fear with SEDA is that I have a pretty high
msg flow at times and if my app goes down and there are messages on the SEDA
queue (that is, pulled off of various jms brokers/queues and waiting on a
SEDA queue) the msgs will be lost. But I wonder, can a transaction work
across that? For example, in a transaction, pull the msgs off of three
different activemq instances and place in SEDA queue, but only consume one
of the msgs from SEDA and then shut down the system. Can the transaction be
maintained so that the two unconsumed msgs roll back to their respective
brokers/queues? I feel some more testing coming on... :) Any thought?
Thanks again, -J
--
View this message in context: http://camel.465427.n5.nabble.com/Best-way-to-consuming-same-queue-from-two-brokers-tp5742100p5742134.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Best way to consuming same queue from two brokers?
Posted by contactreji <co...@gmail.com>.
Hi Dunnlow
If you are using in memory queue, have you thought about using seda: instead
of direct : ?
seda would be able to store message since its asynchronous and can buffer up
data in case other routes dont come up on time .
But direct is highly synchronous. Might have data loss of consumers are not
in sync with it.
Reji
--
View this message in context: http://camel.465427.n5.nabble.com/Best-way-to-consuming-same-queue-from-two-brokers-tp5742100p5742115.html
Sent from the Camel - Users mailing list archive at Nabble.com.