You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Charles Moulliard <cm...@gmail.com> on 2010/02/18 16:52:53 UTC
Can we increase the number of threads of a camel to increase
performances when we must process ordered messages ?
Hi ,
I try to figure out if it is possible or make sense to increase the number
or threads (because we would like to increase performances, ...) of a camel
jms consumer (defined as a endpoint = from) when the messages must be
processed according to their order ?
ex : delivery order of the messages in the queue
message 1, message 2, message3
Message 1 must be processed before 2 and 3 because the information present
in this message must be displayed on the console before the 2 and 3.
If I increase the number of threads of my route, camel will consume message
1, 2 and 3 at the same time but when the result will be delivered to the
last endpoint of the route, the order could be completely different. So I'm
not sure that we can increase performances of the platform by increasing the
number threads in this case !
Kind regards,
Charles Moulliard
Senior Enterprise Architect
Apache Camel Committer
*****************************
blog : http://cmoulliard.blogspot.com
twitter : http://twitter.com/cmoulliard
Linkedlin : http://www.linkedin.com/in/charlesmoulliard
Apache Camel Group :
http://www.linkedin.com/groups?home=&gid=2447439&trk=anet_ug_hm
Re: Can we increase the number of threads of a camel to increase
performances when we must process ordered messages ?
Posted by Charles Moulliard <cm...@gmail.com>.
yeah. Thanks for the explanation
I see the interest to use that now :)
Charles Moulliard
Senior Enterprise Architect
Apache Camel Committer
*****************************
blog : http://cmoulliard.blogspot.com
twitter : http://twitter.com/cmoulliard
Linkedlin : http://www.linkedin.com/in/charlesmoulliard
Apache Camel Group :
http://www.linkedin.com/groups?home=&gid=2447439&trk=anet_ug_hm
On Thu, Feb 18, 2010 at 5:46 PM, Claus Ibsen <cl...@gmail.com> wrote:
> On Thu, Feb 18, 2010 at 5:43 PM, Charles Moulliard <cm...@gmail.com>
> wrote:
> > I had a look to the stream Resequencer example and I have a last question
> :
> >
> > If we have a thread blocking the processing of one of the message, what
> will
> > be the benefit to use a streamResequencer vs consume sequentially
> messages
> > as finally the resequencer has to wait that all the messages have been
> > processed before to send them to the next endpoint ?
> >
>
> The resequencer will only wait until it got the next missing message.
>
> If you need to send messages in the order 1, 2, 3, 4 and 5.
>
> And they arrive as
>
> 3,1 will send 1 as its the first one we need
> 3,4 is in queue now, waiting for 2
> 3,4,2 all will be send as we got 2,3,4 ordered now
> 5 will be send asap
>
>
> >
> > Charles Moulliard
> > Senior Enterprise Architect
> > Apache Camel Committer
> >
> > *****************************
> > blog : http://cmoulliard.blogspot.com
> > twitter : http://twitter.com/cmoulliard
> > Linkedlin : http://www.linkedin.com/in/charlesmoulliard
> >
> > Apache Camel Group :
> > http://www.linkedin.com/groups?home=&gid=2447439&trk=anet_ug_hm
> >
> >
> > On Thu, Feb 18, 2010 at 5:00 PM, Claus Ibsen <cl...@gmail.com>
> wrote:
> >
> >> On Thu, Feb 18, 2010 at 4:52 PM, Charles Moulliard <
> cmoulliard@gmail.com>
> >> wrote:
> >> > Hi ,
> >> >
> >> > I try to figure out if it is possible or make sense to increase the
> >> number
> >> > or threads (because we would like to increase performances, ...) of a
> >> camel
> >> > jms consumer (defined as a endpoint = from) when the messages must be
> >> > processed according to their order ?
> >> >
> >> > ex : delivery order of the messages in the queue
> >> >
> >> > message 1, message 2, message3
> >> >
> >> > Message 1 must be processed before 2 and 3 because the information
> >> present
> >> > in this message must be displayed on the console before the 2 and 3.
> >> >
> >> > If I increase the number of threads of my route, camel will consume
> >> message
> >> > 1, 2 and 3 at the same time but when the result will be delivered to
> the
> >> > last endpoint of the route, the order could be completely different.
> So
> >> I'm
> >> > not sure that we can increase performances of the platform by
> increasing
> >> the
> >> > number threads in this case !
> >> >
> >>
> >> You could resequence the messages before that last step.
> >> http://camel.apache.org/resequencer.html
> >>
> >> Just that processing of the messages can happen out of order and in
> >> parallel then all you need at the end is to re-order them using the
> >> resequencer EIP.
> >> For that you need an unique numbering. I doubt the JMSMessageId can be
> >> used as its just unique but not in ascending order.
> >>
> >> I think the resequencer uses Long for numbering and that they number
> >> be ordered, eg. 1,2,3,4 and so forth.
> >> And its currently in memory only which means the messages are kept in
> >> memory. And it uses another thread to dispath the message outwards,
> >> which means that transactions doesnt work to well.
> >>
> >> In the future the resequencer will have support for a persistent data
> >> store.
> >>
> >>
> >>
> >>
> >> > Kind regards,
> >> >
> >> > Charles Moulliard
> >> > Senior Enterprise Architect
> >> > Apache Camel Committer
> >> >
> >> > *****************************
> >> > blog : http://cmoulliard.blogspot.com
> >> > twitter : http://twitter.com/cmoulliard
> >> > Linkedlin : http://www.linkedin.com/in/charlesmoulliard
> >> >
> >> > Apache Camel Group :
> >> > http://www.linkedin.com/groups?home=&gid=2447439&trk=anet_ug_hm
> >> >
> >>
> >>
> >>
> >> --
> >> Claus Ibsen
> >> Apache Camel Committer
> >>
> >> Author of Camel in Action: http://www.manning.com/ibsen/
> >> Open Source Integration: http://fusesource.com
> >> Blog: http://davsclaus.blogspot.com/
> >> Twitter: http://twitter.com/davsclaus
> >>
> >
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
Re: Can we increase the number of threads of a camel to increase
performances when we must process ordered messages ?
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Feb 18, 2010 at 5:43 PM, Charles Moulliard <cm...@gmail.com> wrote:
> I had a look to the stream Resequencer example and I have a last question :
>
> If we have a thread blocking the processing of one of the message, what will
> be the benefit to use a streamResequencer vs consume sequentially messages
> as finally the resequencer has to wait that all the messages have been
> processed before to send them to the next endpoint ?
>
The resequencer will only wait until it got the next missing message.
If you need to send messages in the order 1, 2, 3, 4 and 5.
And they arrive as
3,1 will send 1 as its the first one we need
3,4 is in queue now, waiting for 2
3,4,2 all will be send as we got 2,3,4 ordered now
5 will be send asap
>
> Charles Moulliard
> Senior Enterprise Architect
> Apache Camel Committer
>
> *****************************
> blog : http://cmoulliard.blogspot.com
> twitter : http://twitter.com/cmoulliard
> Linkedlin : http://www.linkedin.com/in/charlesmoulliard
>
> Apache Camel Group :
> http://www.linkedin.com/groups?home=&gid=2447439&trk=anet_ug_hm
>
>
> On Thu, Feb 18, 2010 at 5:00 PM, Claus Ibsen <cl...@gmail.com> wrote:
>
>> On Thu, Feb 18, 2010 at 4:52 PM, Charles Moulliard <cm...@gmail.com>
>> wrote:
>> > Hi ,
>> >
>> > I try to figure out if it is possible or make sense to increase the
>> number
>> > or threads (because we would like to increase performances, ...) of a
>> camel
>> > jms consumer (defined as a endpoint = from) when the messages must be
>> > processed according to their order ?
>> >
>> > ex : delivery order of the messages in the queue
>> >
>> > message 1, message 2, message3
>> >
>> > Message 1 must be processed before 2 and 3 because the information
>> present
>> > in this message must be displayed on the console before the 2 and 3.
>> >
>> > If I increase the number of threads of my route, camel will consume
>> message
>> > 1, 2 and 3 at the same time but when the result will be delivered to the
>> > last endpoint of the route, the order could be completely different. So
>> I'm
>> > not sure that we can increase performances of the platform by increasing
>> the
>> > number threads in this case !
>> >
>>
>> You could resequence the messages before that last step.
>> http://camel.apache.org/resequencer.html
>>
>> Just that processing of the messages can happen out of order and in
>> parallel then all you need at the end is to re-order them using the
>> resequencer EIP.
>> For that you need an unique numbering. I doubt the JMSMessageId can be
>> used as its just unique but not in ascending order.
>>
>> I think the resequencer uses Long for numbering and that they number
>> be ordered, eg. 1,2,3,4 and so forth.
>> And its currently in memory only which means the messages are kept in
>> memory. And it uses another thread to dispath the message outwards,
>> which means that transactions doesnt work to well.
>>
>> In the future the resequencer will have support for a persistent data
>> store.
>>
>>
>>
>>
>> > Kind regards,
>> >
>> > Charles Moulliard
>> > Senior Enterprise Architect
>> > Apache Camel Committer
>> >
>> > *****************************
>> > blog : http://cmoulliard.blogspot.com
>> > twitter : http://twitter.com/cmoulliard
>> > Linkedlin : http://www.linkedin.com/in/charlesmoulliard
>> >
>> > Apache Camel Group :
>> > http://www.linkedin.com/groups?home=&gid=2447439&trk=anet_ug_hm
>> >
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Can we increase the number of threads of a camel to increase
performances when we must process ordered messages ?
Posted by Charles Moulliard <cm...@gmail.com>.
I had a look to the stream Resequencer example and I have a last question :
If we have a thread blocking the processing of one of the message, what will
be the benefit to use a streamResequencer vs consume sequentially messages
as finally the resequencer has to wait that all the messages have been
processed before to send them to the next endpoint ?
Charles Moulliard
Senior Enterprise Architect
Apache Camel Committer
*****************************
blog : http://cmoulliard.blogspot.com
twitter : http://twitter.com/cmoulliard
Linkedlin : http://www.linkedin.com/in/charlesmoulliard
Apache Camel Group :
http://www.linkedin.com/groups?home=&gid=2447439&trk=anet_ug_hm
On Thu, Feb 18, 2010 at 5:00 PM, Claus Ibsen <cl...@gmail.com> wrote:
> On Thu, Feb 18, 2010 at 4:52 PM, Charles Moulliard <cm...@gmail.com>
> wrote:
> > Hi ,
> >
> > I try to figure out if it is possible or make sense to increase the
> number
> > or threads (because we would like to increase performances, ...) of a
> camel
> > jms consumer (defined as a endpoint = from) when the messages must be
> > processed according to their order ?
> >
> > ex : delivery order of the messages in the queue
> >
> > message 1, message 2, message3
> >
> > Message 1 must be processed before 2 and 3 because the information
> present
> > in this message must be displayed on the console before the 2 and 3.
> >
> > If I increase the number of threads of my route, camel will consume
> message
> > 1, 2 and 3 at the same time but when the result will be delivered to the
> > last endpoint of the route, the order could be completely different. So
> I'm
> > not sure that we can increase performances of the platform by increasing
> the
> > number threads in this case !
> >
>
> You could resequence the messages before that last step.
> http://camel.apache.org/resequencer.html
>
> Just that processing of the messages can happen out of order and in
> parallel then all you need at the end is to re-order them using the
> resequencer EIP.
> For that you need an unique numbering. I doubt the JMSMessageId can be
> used as its just unique but not in ascending order.
>
> I think the resequencer uses Long for numbering and that they number
> be ordered, eg. 1,2,3,4 and so forth.
> And its currently in memory only which means the messages are kept in
> memory. And it uses another thread to dispath the message outwards,
> which means that transactions doesnt work to well.
>
> In the future the resequencer will have support for a persistent data
> store.
>
>
>
>
> > Kind regards,
> >
> > Charles Moulliard
> > Senior Enterprise Architect
> > Apache Camel Committer
> >
> > *****************************
> > blog : http://cmoulliard.blogspot.com
> > twitter : http://twitter.com/cmoulliard
> > Linkedlin : http://www.linkedin.com/in/charlesmoulliard
> >
> > Apache Camel Group :
> > http://www.linkedin.com/groups?home=&gid=2447439&trk=anet_ug_hm
> >
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
Re: Can we increase the number of threads of a camel to increase
performances when we must process ordered messages ?
Posted by Claus Ibsen <cl...@gmail.com>.
And sometimes you can use JMSXGroupID to work with different separated
groups of messages in parallel.
That can improve performance as well.
eg JMSXGroupID can be a stock ID symbol so all messages with the same
symbol are processed in order.
But different stock symbols can be consumed by multiple consumers in parallel.
On Thu, Feb 18, 2010 at 5:00 PM, Claus Ibsen <cl...@gmail.com> wrote:
> On Thu, Feb 18, 2010 at 4:52 PM, Charles Moulliard <cm...@gmail.com> wrote:
>> Hi ,
>>
>> I try to figure out if it is possible or make sense to increase the number
>> or threads (because we would like to increase performances, ...) of a camel
>> jms consumer (defined as a endpoint = from) when the messages must be
>> processed according to their order ?
>>
>> ex : delivery order of the messages in the queue
>>
>> message 1, message 2, message3
>>
>> Message 1 must be processed before 2 and 3 because the information present
>> in this message must be displayed on the console before the 2 and 3.
>>
>> If I increase the number of threads of my route, camel will consume message
>> 1, 2 and 3 at the same time but when the result will be delivered to the
>> last endpoint of the route, the order could be completely different. So I'm
>> not sure that we can increase performances of the platform by increasing the
>> number threads in this case !
>>
>
> You could resequence the messages before that last step.
> http://camel.apache.org/resequencer.html
>
> Just that processing of the messages can happen out of order and in
> parallel then all you need at the end is to re-order them using the
> resequencer EIP.
> For that you need an unique numbering. I doubt the JMSMessageId can be
> used as its just unique but not in ascending order.
>
> I think the resequencer uses Long for numbering and that they number
> be ordered, eg. 1,2,3,4 and so forth.
> And its currently in memory only which means the messages are kept in
> memory. And it uses another thread to dispath the message outwards,
> which means that transactions doesnt work to well.
>
> In the future the resequencer will have support for a persistent data store.
>
>
>
>
>> Kind regards,
>>
>> Charles Moulliard
>> Senior Enterprise Architect
>> Apache Camel Committer
>>
>> *****************************
>> blog : http://cmoulliard.blogspot.com
>> twitter : http://twitter.com/cmoulliard
>> Linkedlin : http://www.linkedin.com/in/charlesmoulliard
>>
>> Apache Camel Group :
>> http://www.linkedin.com/groups?home=&gid=2447439&trk=anet_ug_hm
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Can we increase the number of threads of a camel to increase
performances when we must process ordered messages ?
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Feb 18, 2010 at 4:52 PM, Charles Moulliard <cm...@gmail.com> wrote:
> Hi ,
>
> I try to figure out if it is possible or make sense to increase the number
> or threads (because we would like to increase performances, ...) of a camel
> jms consumer (defined as a endpoint = from) when the messages must be
> processed according to their order ?
>
> ex : delivery order of the messages in the queue
>
> message 1, message 2, message3
>
> Message 1 must be processed before 2 and 3 because the information present
> in this message must be displayed on the console before the 2 and 3.
>
> If I increase the number of threads of my route, camel will consume message
> 1, 2 and 3 at the same time but when the result will be delivered to the
> last endpoint of the route, the order could be completely different. So I'm
> not sure that we can increase performances of the platform by increasing the
> number threads in this case !
>
You could resequence the messages before that last step.
http://camel.apache.org/resequencer.html
Just that processing of the messages can happen out of order and in
parallel then all you need at the end is to re-order them using the
resequencer EIP.
For that you need an unique numbering. I doubt the JMSMessageId can be
used as its just unique but not in ascending order.
I think the resequencer uses Long for numbering and that they number
be ordered, eg. 1,2,3,4 and so forth.
And its currently in memory only which means the messages are kept in
memory. And it uses another thread to dispath the message outwards,
which means that transactions doesnt work to well.
In the future the resequencer will have support for a persistent data store.
> Kind regards,
>
> Charles Moulliard
> Senior Enterprise Architect
> Apache Camel Committer
>
> *****************************
> blog : http://cmoulliard.blogspot.com
> twitter : http://twitter.com/cmoulliard
> Linkedlin : http://www.linkedin.com/in/charlesmoulliard
>
> Apache Camel Group :
> http://www.linkedin.com/groups?home=&gid=2447439&trk=anet_ug_hm
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus