You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by robina <ra...@fomltd.com> on 2016/08/17 13:28:45 UTC
Disabling concurrency in camel route
Hi,
I've encountered a scenario in a camel route involving (I believe)
concurrency:
My route has a netty4 tcp consumer, which then uses a multicast to pass the
message through a number of xsl transformers which then output to ActiveMQ
endpoints.
I've observed that when I receive a large XML message, followed almost
immediately (approx. 1 millisecond) by a smaller XML message, the smaller
message passes through the route before the large message (obviously to do
with the cost of the transformation process). The order of the messages is
important, so I'd like to control this behaviour.
I'm trying to ensure that only one message is processed by the route at any
given time, so would the solution be as simple as setting maxPoolSize=1 on
the netty4 consumer?
I'm having a hard time understanding the threading/concurrency model in
camel, so can anyone help me with a good description somewhere of the
lifecycle of a message through a camel route?
Thank you in advance,
Rob
--
View this message in context: http://camel.465427.n5.nabble.com/Disabling-concurrency-in-camel-route-tp5786531.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Disabling concurrency in camel route
Posted by Brad Johnson <br...@mediadriver.com>.
True enough. I used a resequencer to make sure they went in and out.
<unmarshal ref="recordn" />
<split>
<simple>${body}</simple>
<setHeader headerName="seqnum">
<simple>exchangeProperty.CamelSplitIndex</simple>
</setHeader>
<to uri="seda:processRecordsQueue" />
</split>
...
<resequence>
<stream-config capacity="50000" timeout="15000" />
<simple>in.header.seqnum</simple>
<to uri="seda:outQueue" />
</resequence>
On Thu, Sep 1, 2016 at 2:08 PM, Quinn Stevenson <quinn@pronoia-solutions.com
> wrote:
> I’d be careful with configuring the number of consumers on the SEDA queue
> if you need to preserve order - you’ll likely get stuff processed
> out-of-order.
>
>
> > On Sep 1, 2016, at 1:00 PM, Brad Johnson <br...@mediadriver.com>
> wrote:
> >
> > I'd second what Quinn said. The other nice thing about the SEDA queue is
> > you can directly configure the number of threads that are consuming from
> it.
> >
> > On Thu, Sep 1, 2016 at 1:50 PM, Quinn Stevenson <
> quinn@pronoia-solutions.com
> >> wrote:
> >
> >> When I’ve hit situations like this, I’ve used the SEDA component to
> >> serialize the processing after the message was received. All of the
> >> processing winds up in the SEDA route, and the actual receiving route
> (in
> >> this case, from Netty) just calls the SEDA route. Something like
> >>
> >> <route id=“receiver”>
> >> <from uri=“netty://…..>
> >> <to “seda://do-work>
> >> </route>
> >>
> >> <route id=“serial-processor”>
> >> <from uri=“seda://do-work <seda://do-work>>
> >> <bean id=“processor” method=“process” />
> >> <to uri=“destination://…” />
> >> </route>
> >>
> >>
> >>> On Aug 17, 2016, at 8:47 AM, Vitalii Tymchyshyn <vi...@tym.im> wrote:
> >>>
> >>> You would need a combination.
> >>>
> >>> Ср, 17 серп. 2016 09:41 користувач robina <ra...@fomltd.com> пише:
> >>>
> >>>> Thanks for the reply Vitalii,
> >>>>
> >>>> Would setting synchronous=true on its own ensure that only one message
> >> is
> >>>> processed by the route at any given time? Or is it the combination of
> >> the
> >>>> maxPoolSize of 1 and synchronous=true setting?
> >>>>
> >>>> Regards,
> >>>> Rob
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> View this message in context:
> >>>> http://camel.465427.n5.nabble.com/Disabling-concurrency-in-
> >> camel-route-tp5786531p5786533.html
> >>>> Sent from the Camel - Users mailing list archive at Nabble.com.
> >>>>
> >>
> >>
>
>
Re: Disabling concurrency in camel route
Posted by Brad Johnson <br...@mediadriver.com>.
True enough. I've had to deal with that before by using an aggregator and
assigning sequential numbers to the records as they go through and the
aggregate the results. Obviously that won't work if the records have to be
processed in the exact order they come in and not just written out in the
same order. I did in the case of processing credit cards where the
transactions could go in any order but they wanted the results written back
out in the same order they came in in the input file. One of the files had
a footer that came in and had to be written out as a footer so the
aggregation was important for that.
<unmarshal ref="recordsIn" />
<split>
<simple>${body}</simple>
<setHeader headerName="seqnum">
<simple>exchangeProperty.CamelSplitIndex</simple>
</setHeader>
<to uri="seda:processRecordsQueue" />
</split>
On Thu, Sep 1, 2016 at 2:08 PM, Quinn Stevenson <quinn@pronoia-solutions.com
> wrote:
> I’d be careful with configuring the number of consumers on the SEDA queue
> if you need to preserve order - you’ll likely get stuff processed
> out-of-order.
>
>
> > On Sep 1, 2016, at 1:00 PM, Brad Johnson <br...@mediadriver.com>
> wrote:
> >
> > I'd second what Quinn said. The other nice thing about the SEDA queue is
> > you can directly configure the number of threads that are consuming from
> it.
> >
> > On Thu, Sep 1, 2016 at 1:50 PM, Quinn Stevenson <
> quinn@pronoia-solutions.com
> >> wrote:
> >
> >> When I’ve hit situations like this, I’ve used the SEDA component to
> >> serialize the processing after the message was received. All of the
> >> processing winds up in the SEDA route, and the actual receiving route
> (in
> >> this case, from Netty) just calls the SEDA route. Something like
> >>
> >> <route id=“receiver”>
> >> <from uri=“netty://…..>
> >> <to “seda://do-work>
> >> </route>
> >>
> >> <route id=“serial-processor”>
> >> <from uri=“seda://do-work <seda://do-work>>
> >> <bean id=“processor” method=“process” />
> >> <to uri=“destination://…” />
> >> </route>
> >>
> >>
> >>> On Aug 17, 2016, at 8:47 AM, Vitalii Tymchyshyn <vi...@tym.im> wrote:
> >>>
> >>> You would need a combination.
> >>>
> >>> Ср, 17 серп. 2016 09:41 користувач robina <ra...@fomltd.com> пише:
> >>>
> >>>> Thanks for the reply Vitalii,
> >>>>
> >>>> Would setting synchronous=true on its own ensure that only one message
> >> is
> >>>> processed by the route at any given time? Or is it the combination of
> >> the
> >>>> maxPoolSize of 1 and synchronous=true setting?
> >>>>
> >>>> Regards,
> >>>> Rob
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> View this message in context:
> >>>> http://camel.465427.n5.nabble.com/Disabling-concurrency-in-
> >> camel-route-tp5786531p5786533.html
> >>>> Sent from the Camel - Users mailing list archive at Nabble.com.
> >>>>
> >>
> >>
>
>
Re: Disabling concurrency in camel route
Posted by Quinn Stevenson <qu...@pronoia-solutions.com>.
I’d be careful with configuring the number of consumers on the SEDA queue if you need to preserve order - you’ll likely get stuff processed out-of-order.
> On Sep 1, 2016, at 1:00 PM, Brad Johnson <br...@mediadriver.com> wrote:
>
> I'd second what Quinn said. The other nice thing about the SEDA queue is
> you can directly configure the number of threads that are consuming from it.
>
> On Thu, Sep 1, 2016 at 1:50 PM, Quinn Stevenson <quinn@pronoia-solutions.com
>> wrote:
>
>> When I’ve hit situations like this, I’ve used the SEDA component to
>> serialize the processing after the message was received. All of the
>> processing winds up in the SEDA route, and the actual receiving route (in
>> this case, from Netty) just calls the SEDA route. Something like
>>
>> <route id=“receiver”>
>> <from uri=“netty://…..>
>> <to “seda://do-work>
>> </route>
>>
>> <route id=“serial-processor”>
>> <from uri=“seda://do-work <seda://do-work>>
>> <bean id=“processor” method=“process” />
>> <to uri=“destination://…” />
>> </route>
>>
>>
>>> On Aug 17, 2016, at 8:47 AM, Vitalii Tymchyshyn <vi...@tym.im> wrote:
>>>
>>> You would need a combination.
>>>
>>> Ср, 17 серп. 2016 09:41 користувач robina <ra...@fomltd.com> пише:
>>>
>>>> Thanks for the reply Vitalii,
>>>>
>>>> Would setting synchronous=true on its own ensure that only one message
>> is
>>>> processed by the route at any given time? Or is it the combination of
>> the
>>>> maxPoolSize of 1 and synchronous=true setting?
>>>>
>>>> Regards,
>>>> Rob
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://camel.465427.n5.nabble.com/Disabling-concurrency-in-
>> camel-route-tp5786531p5786533.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>
>>
Re: Disabling concurrency in camel route
Posted by Brad Johnson <br...@mediadriver.com>.
I'd second what Quinn said. The other nice thing about the SEDA queue is
you can directly configure the number of threads that are consuming from it.
On Thu, Sep 1, 2016 at 1:50 PM, Quinn Stevenson <quinn@pronoia-solutions.com
> wrote:
> When I’ve hit situations like this, I’ve used the SEDA component to
> serialize the processing after the message was received. All of the
> processing winds up in the SEDA route, and the actual receiving route (in
> this case, from Netty) just calls the SEDA route. Something like
>
> <route id=“receiver”>
> <from uri=“netty://…..>
> <to “seda://do-work>
> </route>
>
> <route id=“serial-processor”>
> <from uri=“seda://do-work <seda://do-work>>
> <bean id=“processor” method=“process” />
> <to uri=“destination://…” />
> </route>
>
>
> > On Aug 17, 2016, at 8:47 AM, Vitalii Tymchyshyn <vi...@tym.im> wrote:
> >
> > You would need a combination.
> >
> > Ср, 17 серп. 2016 09:41 користувач robina <ra...@fomltd.com> пише:
> >
> >> Thanks for the reply Vitalii,
> >>
> >> Would setting synchronous=true on its own ensure that only one message
> is
> >> processed by the route at any given time? Or is it the combination of
> the
> >> maxPoolSize of 1 and synchronous=true setting?
> >>
> >> Regards,
> >> Rob
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >> http://camel.465427.n5.nabble.com/Disabling-concurrency-in-
> camel-route-tp5786531p5786533.html
> >> Sent from the Camel - Users mailing list archive at Nabble.com.
> >>
>
>
Re: Disabling concurrency in camel route
Posted by Quinn Stevenson <qu...@pronoia-solutions.com>.
When I’ve hit situations like this, I’ve used the SEDA component to serialize the processing after the message was received. All of the processing winds up in the SEDA route, and the actual receiving route (in this case, from Netty) just calls the SEDA route. Something like
<route id=“receiver”>
<from uri=“netty://…..>
<to “seda://do-work>
</route>
<route id=“serial-processor”>
<from uri=“seda://do-work <seda://do-work>>
<bean id=“processor” method=“process” />
<to uri=“destination://…” />
</route>
> On Aug 17, 2016, at 8:47 AM, Vitalii Tymchyshyn <vi...@tym.im> wrote:
>
> You would need a combination.
>
> Ср, 17 серп. 2016 09:41 користувач robina <ra...@fomltd.com> пише:
>
>> Thanks for the reply Vitalii,
>>
>> Would setting synchronous=true on its own ensure that only one message is
>> processed by the route at any given time? Or is it the combination of the
>> maxPoolSize of 1 and synchronous=true setting?
>>
>> Regards,
>> Rob
>>
>>
>>
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/Disabling-concurrency-in-camel-route-tp5786531p5786533.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
Re: Disabling concurrency in camel route
Posted by Vitalii Tymchyshyn <vi...@tym.im>.
You would need a combination.
Ср, 17 серп. 2016 09:41 користувач robina <ra...@fomltd.com> пише:
> Thanks for the reply Vitalii,
>
> Would setting synchronous=true on its own ensure that only one message is
> processed by the route at any given time? Or is it the combination of the
> maxPoolSize of 1 and synchronous=true setting?
>
> Regards,
> Rob
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Disabling-concurrency-in-camel-route-tp5786531p5786533.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Disabling concurrency in camel route
Posted by Karts <ka...@gmail.com>.
I believe you can just set .threads(1)
--
View this message in context: http://camel.465427.n5.nabble.com/Disabling-concurrency-in-camel-route-tp5786531p5786535.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Disabling concurrency in camel route
Posted by robina <ra...@fomltd.com>.
Thanks for the reply Vitalii,
Would setting synchronous=true on its own ensure that only one message is
processed by the route at any given time? Or is it the combination of the
maxPoolSize of 1 and synchronous=true setting?
Regards,
Rob
--
View this message in context: http://camel.465427.n5.nabble.com/Disabling-concurrency-in-camel-route-tp5786531p5786533.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Disabling concurrency in camel route
Posted by Vitalii Tymchyshyn <vi...@tym.im>.
You should also do "synchronous=true". Otherwise you may experience
reordering on any asynchronous producer.
Another option is to use http://camel.apache.org/resequencer.html when you
want the sequence restored.
Best regards, Vitalii Tymchyshyn
Ср, 17 серп. 2016 о 09:28 robina <ra...@fomltd.com> пише:
> Hi,
>
> I've encountered a scenario in a camel route involving (I believe)
> concurrency:
>
> My route has a netty4 tcp consumer, which then uses a multicast to pass the
> message through a number of xsl transformers which then output to ActiveMQ
> endpoints.
>
> I've observed that when I receive a large XML message, followed almost
> immediately (approx. 1 millisecond) by a smaller XML message, the smaller
> message passes through the route before the large message (obviously to do
> with the cost of the transformation process). The order of the messages is
> important, so I'd like to control this behaviour.
>
> I'm trying to ensure that only one message is processed by the route at any
> given time, so would the solution be as simple as setting maxPoolSize=1 on
> the netty4 consumer?
>
> I'm having a hard time understanding the threading/concurrency model in
> camel, so can anyone help me with a good description somewhere of the
> lifecycle of a message through a camel route?
>
> Thank you in advance,
> Rob
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Disabling-concurrency-in-camel-route-tp5786531.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>