You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by slew <sl...@googlemail.com> on 2010/10/14 22:05:05 UTC
Cross context connections, SEDA and BlockQueues
Hi,
In our application we have a routing slip that is configured at runtime to
invoke endpoints from "plugin" camel routes (camel 2.3) that are provided in
a separate camel context. We are using smx 3, so these plugins are packaged
as separate JBI assemblies.
I'm trying to work out the best method to communicate between these
contexts. The original way I used was with the VM component. For
performance, I had to set the concurrentConsumers property, but this has a
drawback in that the consumer threads are fixed and so as the number of
plugins increases the number of waiting threads increases.
I've been looking at replacing this with JBI, as I'm currently using SMX3.
This has thrown up some other questions, but as we'd look to move to SMX4 at
some point (possibly soon) and JBI is not really the future of SMX, I'm
wondering if there are better alternatives.
Going back to the VM solution. The documentation has the following:
> Be aware that adding a thread pool to a SEDA endpoint by doing something
> like:
> from("seda:stageName").thread(5).process(...)
> Can wind up with two BlockQueues: one from the SEDA endpoint, and one from
> the workqueue of the thread pool, which may not be what you want.
I'm not sure exactly what this means, could someone explain it to me please
- really sorry if it's a thick question? I assume it's not saying there's a
potential for a dead lock, but I can't work out what else it means. Would
this be the correct approach in my scenario, i.e. so that more threads are
working when the route is busy, less when it's not?
Is there a better solution to this cross-context connections, e.g. JMS,
ActiveMQ. I like the VM approach as it works nicely with exception
propagation.
I'd really appreciate anyones advice on this. Please let me know if you
need me to clarify anything.
Thanks,
Steve.
--
View this message in context: http://camel.465427.n5.nabble.com/Cross-context-connections-SEDA-and-BlockQueues-tp3212752p3212752.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Cross context connections, SEDA and BlockQueues
Posted by slew <sl...@googlemail.com>.
Thanks for taking the time to answer Stan. I'll have another go at trying to
get camel to convert the exception. I have a snapshot of the 2010.02
servicemix-camel component which I think handles the jbi part, I'll try
updating to the latest to make sure I haven't just got a bug in my version.
Stan Lewis wrote:
>
> I believe the servicemix-camel component handles this already, i.e.
> it'll convert an exception to a JBI fault and send it back as a
> response, assuming the MEP allows for it.
>
> On Thu, Oct 14, 2010 at 4:55 PM, slew <sl...@googlemail.com> wrote:
>>
>> Hi Stan,
>>
>> Thanks for the response and clarifying the blocking queue question, makes
>> perfect sense now.
>>
>> I feel more comfortable about continuing with the JBI endpoints now, so
>> thanks again. One difficulty I'm having (i've posted this on the smx
>> forum
>> today, so sorry if this counts as a repost), but I'm not sure about the
>> recommended way to propagate exceptions using the jbi component in camel.
>> As the exchange goes through the NMR, I guess the exceptions need to be
>> marshalled somehow, but I'm not sure of the recommended way to do this.
>> Should I trap the exceptions and convert them to a JBIFault manually or
>> should I let the camel component do some of this? I've tried setting the
>> convertException flag, but don't see any difference and in both cases
>> lose
>> the original exception. Alternatively, I can set a property on the
>> exchange
>> containing the exception and set the exchange as faulted, or turn the
>> exception into xml and propagate it that way as the fault body, but I
>> want
>> to make sure I'm not fighting the intended way of doing things and
>> missing
>> the recommended approach.
>>
>> Thanks for any more advice,
>> Steve.
>>
>>
>>
>> Stan Lewis wrote:
>>>
>>> Hi Steve,
>>>
>>> On Thu, Oct 14, 2010 at 4:05 PM, slew <sl...@googlemail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> In our application we have a routing slip that is configured at runtime
>>>> to
>>>> invoke endpoints from "plugin" camel routes (camel 2.3) that are
>>>> provided
>>>> in
>>>> a separate camel context. We are using smx 3, so these plugins are
>>>> packaged
>>>> as separate JBI assemblies.
>>>>
>>>> I'm trying to work out the best method to communicate between these
>>>> contexts. The original way I used was with the VM component. For
>>>> performance, I had to set the concurrentConsumers property, but this
>>>> has
>>>> a
>>>> drawback in that the consumer threads are fixed and so as the number of
>>>> plugins increases the number of waiting threads increases.
>>>>
>>>> I've been looking at replacing this with JBI, as I'm currently using
>>>> SMX3.
>>>> This has thrown up some other questions, but as we'd look to move to
>>>> SMX4
>>>> at
>>>> some point (possibly soon) and JBI is not really the future of SMX, I'm
>>>> wondering if there are better alternatives.
>>>
>>> You can use JBI in either SMX3 or SMX4 so it's a good migration path,
>>> i.e. you should be able to continue using JBI endpoints when moving to
>>> SMX4 and then you can migrate them to something else. For new stuff
>>> in SMX4 you can use the NMR component -
>>> http://camel.apache.org/nmr.html which I think might be better suited
>>> in an OSGi environment than the VM transport. And also you can always
>>> use JMS as well, probably more useful if you want to persist messages
>>> between routes to avoid losing messages.
>>>
>>>> Going back to the VM solution. The documentation has the following:
>>>>
>>>>> Be aware that adding a thread pool to a SEDA endpoint by doing
>>>>> something
>>>>> like:
>>>>> from("seda:stageName").thread(5).process(...)
>>>>> Can wind up with two BlockQueues: one from the SEDA endpoint, and one
>>>>> from
>>>>> the workqueue of the thread pool, which may not be what you want.
>>>>
>>>> I'm not sure exactly what this means, could someone explain it to me
>>>> please
>>>> - really sorry if it's a thick question? I assume it's not saying
>>>> there's a
>>>> potential for a dead lock, but I can't work out what else it means.
>>>> Would
>>>> this be the correct approach in my scenario, i.e. so that more threads
>>>> are
>>>> working when the route is busy, less when it's not?
>>>
>>> I think it's a little typo, it should be BlockingQueues, meaning that
>>> a message would first go into the SEDA BlockingQueue, then wind up in
>>> the thread pool's BlockingQueue, so would introduce a performance
>>> impact.
>>>
>>>> Is there a better solution to this cross-context connections, e.g. JMS,
>>>> ActiveMQ. I like the VM approach as it works nicely with exception
>>>> propagation.
>>>>
>>>> I'd really appreciate anyones advice on this. Please let me know if
>>>> you
>>>> need me to clarify anything.
>>>>
>>>> Thanks,
>>>> Steve.
>>>> --
>>>> View this message in context:
>>>> http://camel.465427.n5.nabble.com/Cross-context-connections-SEDA-and-BlockQueues-tp3212752p3212752.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>
>>>
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/Cross-context-connections-SEDA-and-BlockQueues-tp3212752p3212817.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>
>
--
View this message in context: http://camel.465427.n5.nabble.com/Cross-context-connections-SEDA-and-BlockQueues-tp3212752p3213809.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Cross context connections, SEDA and BlockQueues
Posted by Stan Lewis <sl...@fusesource.com>.
I believe the servicemix-camel component handles this already, i.e.
it'll convert an exception to a JBI fault and send it back as a
response, assuming the MEP allows for it.
On Thu, Oct 14, 2010 at 4:55 PM, slew <sl...@googlemail.com> wrote:
>
> Hi Stan,
>
> Thanks for the response and clarifying the blocking queue question, makes
> perfect sense now.
>
> I feel more comfortable about continuing with the JBI endpoints now, so
> thanks again. One difficulty I'm having (i've posted this on the smx forum
> today, so sorry if this counts as a repost), but I'm not sure about the
> recommended way to propagate exceptions using the jbi component in camel.
> As the exchange goes through the NMR, I guess the exceptions need to be
> marshalled somehow, but I'm not sure of the recommended way to do this.
> Should I trap the exceptions and convert them to a JBIFault manually or
> should I let the camel component do some of this? I've tried setting the
> convertException flag, but don't see any difference and in both cases lose
> the original exception. Alternatively, I can set a property on the exchange
> containing the exception and set the exchange as faulted, or turn the
> exception into xml and propagate it that way as the fault body, but I want
> to make sure I'm not fighting the intended way of doing things and missing
> the recommended approach.
>
> Thanks for any more advice,
> Steve.
>
>
>
> Stan Lewis wrote:
>>
>> Hi Steve,
>>
>> On Thu, Oct 14, 2010 at 4:05 PM, slew <sl...@googlemail.com> wrote:
>>>
>>> Hi,
>>>
>>> In our application we have a routing slip that is configured at runtime
>>> to
>>> invoke endpoints from "plugin" camel routes (camel 2.3) that are provided
>>> in
>>> a separate camel context. We are using smx 3, so these plugins are
>>> packaged
>>> as separate JBI assemblies.
>>>
>>> I'm trying to work out the best method to communicate between these
>>> contexts. The original way I used was with the VM component. For
>>> performance, I had to set the concurrentConsumers property, but this has
>>> a
>>> drawback in that the consumer threads are fixed and so as the number of
>>> plugins increases the number of waiting threads increases.
>>>
>>> I've been looking at replacing this with JBI, as I'm currently using
>>> SMX3.
>>> This has thrown up some other questions, but as we'd look to move to SMX4
>>> at
>>> some point (possibly soon) and JBI is not really the future of SMX, I'm
>>> wondering if there are better alternatives.
>>
>> You can use JBI in either SMX3 or SMX4 so it's a good migration path,
>> i.e. you should be able to continue using JBI endpoints when moving to
>> SMX4 and then you can migrate them to something else. For new stuff
>> in SMX4 you can use the NMR component -
>> http://camel.apache.org/nmr.html which I think might be better suited
>> in an OSGi environment than the VM transport. And also you can always
>> use JMS as well, probably more useful if you want to persist messages
>> between routes to avoid losing messages.
>>
>>> Going back to the VM solution. The documentation has the following:
>>>
>>>> Be aware that adding a thread pool to a SEDA endpoint by doing something
>>>> like:
>>>> from("seda:stageName").thread(5).process(...)
>>>> Can wind up with two BlockQueues: one from the SEDA endpoint, and one
>>>> from
>>>> the workqueue of the thread pool, which may not be what you want.
>>>
>>> I'm not sure exactly what this means, could someone explain it to me
>>> please
>>> - really sorry if it's a thick question? I assume it's not saying
>>> there's a
>>> potential for a dead lock, but I can't work out what else it means.
>>> Would
>>> this be the correct approach in my scenario, i.e. so that more threads
>>> are
>>> working when the route is busy, less when it's not?
>>
>> I think it's a little typo, it should be BlockingQueues, meaning that
>> a message would first go into the SEDA BlockingQueue, then wind up in
>> the thread pool's BlockingQueue, so would introduce a performance
>> impact.
>>
>>> Is there a better solution to this cross-context connections, e.g. JMS,
>>> ActiveMQ. I like the VM approach as it works nicely with exception
>>> propagation.
>>>
>>> I'd really appreciate anyones advice on this. Please let me know if you
>>> need me to clarify anything.
>>>
>>> Thanks,
>>> Steve.
>>> --
>>> View this message in context:
>>> http://camel.465427.n5.nabble.com/Cross-context-connections-SEDA-and-BlockQueues-tp3212752p3212752.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>
>>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Cross-context-connections-SEDA-and-BlockQueues-tp3212752p3212817.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Cross context connections, SEDA and BlockQueues
Posted by slew <sl...@googlemail.com>.
Hi Stan,
Thanks for the response and clarifying the blocking queue question, makes
perfect sense now.
I feel more comfortable about continuing with the JBI endpoints now, so
thanks again. One difficulty I'm having (i've posted this on the smx forum
today, so sorry if this counts as a repost), but I'm not sure about the
recommended way to propagate exceptions using the jbi component in camel.
As the exchange goes through the NMR, I guess the exceptions need to be
marshalled somehow, but I'm not sure of the recommended way to do this.
Should I trap the exceptions and convert them to a JBIFault manually or
should I let the camel component do some of this? I've tried setting the
convertException flag, but don't see any difference and in both cases lose
the original exception. Alternatively, I can set a property on the exchange
containing the exception and set the exchange as faulted, or turn the
exception into xml and propagate it that way as the fault body, but I want
to make sure I'm not fighting the intended way of doing things and missing
the recommended approach.
Thanks for any more advice,
Steve.
Stan Lewis wrote:
>
> Hi Steve,
>
> On Thu, Oct 14, 2010 at 4:05 PM, slew <sl...@googlemail.com> wrote:
>>
>> Hi,
>>
>> In our application we have a routing slip that is configured at runtime
>> to
>> invoke endpoints from "plugin" camel routes (camel 2.3) that are provided
>> in
>> a separate camel context. We are using smx 3, so these plugins are
>> packaged
>> as separate JBI assemblies.
>>
>> I'm trying to work out the best method to communicate between these
>> contexts. The original way I used was with the VM component. For
>> performance, I had to set the concurrentConsumers property, but this has
>> a
>> drawback in that the consumer threads are fixed and so as the number of
>> plugins increases the number of waiting threads increases.
>>
>> I've been looking at replacing this with JBI, as I'm currently using
>> SMX3.
>> This has thrown up some other questions, but as we'd look to move to SMX4
>> at
>> some point (possibly soon) and JBI is not really the future of SMX, I'm
>> wondering if there are better alternatives.
>
> You can use JBI in either SMX3 or SMX4 so it's a good migration path,
> i.e. you should be able to continue using JBI endpoints when moving to
> SMX4 and then you can migrate them to something else. For new stuff
> in SMX4 you can use the NMR component -
> http://camel.apache.org/nmr.html which I think might be better suited
> in an OSGi environment than the VM transport. And also you can always
> use JMS as well, probably more useful if you want to persist messages
> between routes to avoid losing messages.
>
>> Going back to the VM solution. The documentation has the following:
>>
>>> Be aware that adding a thread pool to a SEDA endpoint by doing something
>>> like:
>>> from("seda:stageName").thread(5).process(...)
>>> Can wind up with two BlockQueues: one from the SEDA endpoint, and one
>>> from
>>> the workqueue of the thread pool, which may not be what you want.
>>
>> I'm not sure exactly what this means, could someone explain it to me
>> please
>> - really sorry if it's a thick question? I assume it's not saying
>> there's a
>> potential for a dead lock, but I can't work out what else it means.
>> Would
>> this be the correct approach in my scenario, i.e. so that more threads
>> are
>> working when the route is busy, less when it's not?
>
> I think it's a little typo, it should be BlockingQueues, meaning that
> a message would first go into the SEDA BlockingQueue, then wind up in
> the thread pool's BlockingQueue, so would introduce a performance
> impact.
>
>> Is there a better solution to this cross-context connections, e.g. JMS,
>> ActiveMQ. I like the VM approach as it works nicely with exception
>> propagation.
>>
>> I'd really appreciate anyones advice on this. Please let me know if you
>> need me to clarify anything.
>>
>> Thanks,
>> Steve.
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/Cross-context-connections-SEDA-and-BlockQueues-tp3212752p3212752.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>
>
--
View this message in context: http://camel.465427.n5.nabble.com/Cross-context-connections-SEDA-and-BlockQueues-tp3212752p3212817.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Cross context connections, SEDA and BlockQueues
Posted by Stan Lewis <sl...@fusesource.com>.
Hi Steve,
On Thu, Oct 14, 2010 at 4:05 PM, slew <sl...@googlemail.com> wrote:
>
> Hi,
>
> In our application we have a routing slip that is configured at runtime to
> invoke endpoints from "plugin" camel routes (camel 2.3) that are provided in
> a separate camel context. We are using smx 3, so these plugins are packaged
> as separate JBI assemblies.
>
> I'm trying to work out the best method to communicate between these
> contexts. The original way I used was with the VM component. For
> performance, I had to set the concurrentConsumers property, but this has a
> drawback in that the consumer threads are fixed and so as the number of
> plugins increases the number of waiting threads increases.
>
> I've been looking at replacing this with JBI, as I'm currently using SMX3.
> This has thrown up some other questions, but as we'd look to move to SMX4 at
> some point (possibly soon) and JBI is not really the future of SMX, I'm
> wondering if there are better alternatives.
You can use JBI in either SMX3 or SMX4 so it's a good migration path,
i.e. you should be able to continue using JBI endpoints when moving to
SMX4 and then you can migrate them to something else. For new stuff
in SMX4 you can use the NMR component -
http://camel.apache.org/nmr.html which I think might be better suited
in an OSGi environment than the VM transport. And also you can always
use JMS as well, probably more useful if you want to persist messages
between routes to avoid losing messages.
> Going back to the VM solution. The documentation has the following:
>
>> Be aware that adding a thread pool to a SEDA endpoint by doing something
>> like:
>> from("seda:stageName").thread(5).process(...)
>> Can wind up with two BlockQueues: one from the SEDA endpoint, and one from
>> the workqueue of the thread pool, which may not be what you want.
>
> I'm not sure exactly what this means, could someone explain it to me please
> - really sorry if it's a thick question? I assume it's not saying there's a
> potential for a dead lock, but I can't work out what else it means. Would
> this be the correct approach in my scenario, i.e. so that more threads are
> working when the route is busy, less when it's not?
I think it's a little typo, it should be BlockingQueues, meaning that
a message would first go into the SEDA BlockingQueue, then wind up in
the thread pool's BlockingQueue, so would introduce a performance
impact.
> Is there a better solution to this cross-context connections, e.g. JMS,
> ActiveMQ. I like the VM approach as it works nicely with exception
> propagation.
>
> I'd really appreciate anyones advice on this. Please let me know if you
> need me to clarify anything.
>
> Thanks,
> Steve.
> --
> View this message in context: http://camel.465427.n5.nabble.com/Cross-context-connections-SEDA-and-BlockQueues-tp3212752p3212752.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>