You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by souciance <so...@gmail.com> on 2016/02/22 17:57:11 UTC
Re: Consuming messages from hundred of AMQ queues using camel
routes builder
Hi Darwish
Do you need to do anythng different with the messages? For example do
you require to implement each customer in a different way? Or is it
just that you can messages from 100s of queues but the routebuilder
logic is the same for all?
Best
Souciance
On Mon, Feb 22, 2016 at 5:53 PM, Darwish [via Camel]
<ml...@n5.nabble.com> wrote:
> Hi ,
> My requirement is to process a messages come from many queues each represent
> one customer communication channel , the message payload and the
> processing workflow is identical , to solve this problem i build route
> builder and dynamically set "from" point and attached with a queue name
> and the "to" point to direct point route .
> example
> from uri="actimvemq:queue:customerA
> to uri=direct:processMsg
> Things are OK , my concern is the number of dynamic routes is linearly
> increased with number of customers , if i have 100 cutomers i need to build
> 100 routes ..
>
> any ideas to overcome this issue ?
> Othman Darwish
> ProgressSoft Corp.
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://camel.465427.n5.nabble.com/Consuming-messages-from-hundred-of-AMQ-queues-using-camel-routes-builder-tp5778059.html
> To start a new topic under Camel - Users, email
> ml-node+s465427n465428h31@n5.nabble.com
> To unsubscribe from Camel - Users, click here.
> NAML
--
View this message in context: http://camel.465427.n5.nabble.com/Re-Consuming-messages-from-hundred-of-AMQ-queues-using-camel-routes-builder-tp5778060.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Consuming messages from hundred of AMQ queues using camel
routes builder
Posted by Darwish <ot...@progressoft.com>.
Thanks Stevenson suggestion.... I POC it and the number of routes decreased
dramatically ... enabling concurrent consumer of that route camel AMQ
component will achieve parallel processing ...
Thanks again
-----
Othman Darwish
ProgressSoft Corp.
--
View this message in context: http://camel.465427.n5.nabble.com/Re-Consuming-messages-from-hundred-of-AMQ-queues-using-camel-routes-builder-tp5778060p5778074.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Consuming messages from hundred of AMQ queues using camel routes builder
Posted by Quinn Stevenson <qu...@pronoia-solutions.com>.
Yes - that’s what I’m suggesting.
Also, if all of the messages from the various outward.message.> queues are just forwarded to the same internal queue for processing, you could do this in the broker with a Composite Destination (http://activemq.apache.org/nms/activemq-virtual-destinations.html <http://activemq.apache.org/nms/activemq-virtual-destinations.html>) and avoid some of the overhead of the camel route.
> On Feb 22, 2016, at 10:50 AM, Darwish <ot...@progressoft.com> wrote:
>
> Hi Quinn Stevenson
>
> you mean in my route builder instead of specifying full name of the queue
> that belong to a specific customer , i could use for example
>
> from queue:outward.message.> , then the route will consume all messages
> that holed in all queues as long as it start with "outward.message." , if
> this what you mean this will be great
>
>
>
>
>
>
> -----
> Othman Darwish
> ProgressSoft Corp.
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Re-Consuming-messages-from-hundred-of-AMQ-queues-using-camel-routes-builder-tp5778060p5778069.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Consuming messages from hundred of AMQ queues using camel
routes builder
Posted by Darwish <ot...@progressoft.com>.
Hi Quinn Stevenson
you mean in my route builder instead of specifying full name of the queue
that belong to a specific customer , i could use for example
from queue:outward.message.> , then the route will consume all messages
that holed in all queues as long as it start with "outward.message." , if
this what you mean this will be great
-----
Othman Darwish
ProgressSoft Corp.
--
View this message in context: http://camel.465427.n5.nabble.com/Re-Consuming-messages-from-hundred-of-AMQ-queues-using-camel-routes-builder-tp5778060p5778069.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Consuming messages from hundred of AMQ queues using camel routes builder
Posted by Quinn Stevenson <qu...@pronoia-solutions.com>.
Could you use a wildcard subscription on the initial consuming route?
> On Feb 22, 2016, at 10:20 AM, Darwish <ot...@progressoft.com> wrote:
>
> what i looking for is to find a way of reliable message processing by
> staging my processing as flowing :
>
> single or bounded number of threads ( routes) that initially consume those
> message and push it to another internal staging queue ,and then camel route
> with multiple consumers let say 10 or 20 that work on that internal queue .
> my problem is that camel route does not allow consuming multiple "from
> point" although this is possible when producing a messages ( percipient
> list )
>
>
>
>
> -----
> Othman Darwish
> ProgressSoft Corp.
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Re-Consuming-messages-from-hundred-of-AMQ-queues-using-camel-routes-builder-tp5778060p5778064.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Consuming messages from hundred of AMQ queues using camel
routes builder
Posted by Darwish <ot...@progressoft.com>.
what i looking for is to find a way of reliable message processing by
staging my processing as flowing :
single or bounded number of threads ( routes) that initially consume those
message and push it to another internal staging queue ,and then camel route
with multiple consumers let say 10 or 20 that work on that internal queue .
my problem is that camel route does not allow consuming multiple "from
point" although this is possible when producing a messages ( percipient
list )
-----
Othman Darwish
ProgressSoft Corp.
--
View this message in context: http://camel.465427.n5.nabble.com/Re-Consuming-messages-from-hundred-of-AMQ-queues-using-camel-routes-builder-tp5778060p5778064.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Consuming messages from hundred of AMQ queues using camel
routes builder
Posted by Darwish <ot...@progressoft.com>.
Thanks for suggestion , unfortunately centralized queue not options .. also
i have SAL with each customer and this will make some kind of bottleneck in
processing those messages ...
-----
Othman Darwish
ProgressSoft Corp.
--
View this message in context: http://camel.465427.n5.nabble.com/Re-Consuming-messages-from-hundred-of-AMQ-queues-using-camel-routes-builder-tp5778060p5778063.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Consuming messages from hundred of AMQ queues using camel
routes builder
Posted by souciance <so...@gmail.com>.
Since the data and logic seems to be the same, wouldn't it be possible
to ask all customers to put the messages on the same queue? Perhaps
seperate them via some header value? Makes it easier to handle from
activemq and camel side?
On Mon, Feb 22, 2016 at 6:00 PM, Darwish [via Camel]
<ml...@n5.nabble.com> wrote:
> Hi souciance
>
> The logic is the same
> Othman Darwish
> ProgressSoft Corp.
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://camel.465427.n5.nabble.com/Re-Consuming-messages-from-hundred-of-AMQ-queues-using-camel-routes-builder-tp5778060p5778061.html
> To start a new topic under Camel - Users, email
> ml-node+s465427n465428h31@n5.nabble.com
> To unsubscribe from Camel - Users, click here.
> NAML
--
View this message in context: http://camel.465427.n5.nabble.com/Re-Consuming-messages-from-hundred-of-AMQ-queues-using-camel-routes-builder-tp5778060p5778062.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Consuming messages from hundred of AMQ queues using camel
routes builder
Posted by Darwish <ot...@progressoft.com>.
Hi souciance
The logic is the same
-----
Othman Darwish
ProgressSoft Corp.
--
View this message in context: http://camel.465427.n5.nabble.com/Re-Consuming-messages-from-hundred-of-AMQ-queues-using-camel-routes-builder-tp5778060p5778061.html
Sent from the Camel - Users mailing list archive at Nabble.com.