You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Raul Kripalani <ra...@apache.org> on 2016/03/29 01:18:35 UTC
Multicast version 2
Camel riders,
I'm seeing an issue caused by the MulticastProcessor not handling async
routing end-to-end. It does use its own threadpool to process its routing
tasks, as well as to perform the aggregation (on-the-fly).
However, during all that time, the calling thread is blocked waiting for
the multicast to finish, even if the previous processors passed along an
AsyncCallback. This behaviour basically nullifies the async routing engine.
* How tough do you think it'll be to make the Multicast processor
async-aware end-to-end? My idea is to implement a technique similar to the
TemporaryReplyQueueManager in camel-jms, where we do bookkeeping in an
in-memory data structure, and trigger the AsyncCallback when we've
collected all aggregation steps, or if a timeout triggers.
* Do you think it's worthwhile to dedicate a full thread per request to
process its aggregations? In I/O bound cases, such as waiting for a WS
reply, file download, etc. it feels wasteful to keep a thread parked for
aggregations.
* What do you think about calling the aggregation strategy from the
thread(s) where the response(s) arrive(s)? (which could be from the
multicast threadpool or a component thread e.g. TempReplyQueueManager in
the case of JMS, depending on whether the component is sync or async).
In essence, I'm considering a multicast v2 implementation (which would also
affect RecipientList, Splitter, etc.) for 2.18, allowing the user to choose
the implementation via DSL, e.g.
.multicast().v2()
It it's more efficient and feedback from users is good, we could make it
the default multicast processor in Camel 3.0.
Thoughts?
Cheers,
*Raúl Kripalani*
PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
Messaging Engineer
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
Blog: raul.io | twitter: @raulvk <https://twitter.com/raulvk>
Re: Multicast version 2
Posted by Aaron Whiteside <aw...@mGage.com>.
What do you think of this approach?
MulticastProcessor should pass in it’s own AsyncCallback (or attach a Synchronization if you perfer) where on done() we submit the Exchange to be aggregated. That way if the route is async no threads are blocked waiting for a response, essentially it would be event driven.
Regards,
Aaron
Aaron Whiteside | Director, Technical Architecture
e awhiteside@mGage.com
mGage | The Mobile Engagement Company
w +1.310.846.9269 m
On 3/28/16, 4:18 PM, "Raul Kripalani" <ra...@apache.org> wrote:
>Camel riders,
>
>I'm seeing an issue caused by the MulticastProcessor not handling async
>routing end-to-end. It does use its own threadpool to process its routing
>tasks, as well as to perform the aggregation (on-the-fly).
>
>However, during all that time, the calling thread is blocked waiting for
>the multicast to finish, even if the previous processors passed along an
>AsyncCallback. This behaviour basically nullifies the async routing engine.
>
>* How tough do you think it'll be to make the Multicast processor
>async-aware end-to-end? My idea is to implement a technique similar to the
>TemporaryReplyQueueManager in camel-jms, where we do bookkeeping in an
>in-memory data structure, and trigger the AsyncCallback when we've
>collected all aggregation steps, or if a timeout triggers.
>
>* Do you think it's worthwhile to dedicate a full thread per request to
>process its aggregations? In I/O bound cases, such as waiting for a WS
>reply, file download, etc. it feels wasteful to keep a thread parked for
>aggregations.
>
>* What do you think about calling the aggregation strategy from the
>thread(s) where the response(s) arrive(s)? (which could be from the
>multicast threadpool or a component thread e.g. TempReplyQueueManager in
>the case of JMS, depending on whether the component is sync or async).
>
>In essence, I'm considering a multicast v2 implementation (which would also
>affect RecipientList, Splitter, etc.) for 2.18, allowing the user to choose
>the implementation via DSL, e.g.
>
> .multicast().v2()
>
>It it's more efficient and feedback from users is good, we could make it
>the default multicast processor in Camel 3.0.
>
>Thoughts?
>
>Cheers,
>
>*Raúl Kripalani*
>PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
>Messaging Engineer
>http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
>Blog: raul.io | twitter: @raulvk <https://twitter.com/raulvk>
Re: Multicast version 2
Posted by Aaron Whiteside <aw...@mGage.com>.
What do you think of this approach?
MulticastProcessor should pass in it’s own AsyncCallback (or attach a Synchronization if you perfer) where on done() we submit the Exchange to be aggregated. That way if the route is async no threads are blocked waiting for a response, essentially it would be event driven.
Regards,
Aaron
Aaron Whiteside | Director, Technical Architecture
e awhiteside@mGage.com
mGage | The Mobile Engagement Company
w +1.310.846.9269 m
On 3/28/16, 4:18 PM, "Raul Kripalani" <ra...@apache.org> wrote:
>Camel riders,
>
>I'm seeing an issue caused by the MulticastProcessor not handling async
>routing end-to-end. It does use its own threadpool to process its routing
>tasks, as well as to perform the aggregation (on-the-fly).
>
>However, during all that time, the calling thread is blocked waiting for
>the multicast to finish, even if the previous processors passed along an
>AsyncCallback. This behaviour basically nullifies the async routing engine.
>
>* How tough do you think it'll be to make the Multicast processor
>async-aware end-to-end? My idea is to implement a technique similar to the
>TemporaryReplyQueueManager in camel-jms, where we do bookkeeping in an
>in-memory data structure, and trigger the AsyncCallback when we've
>collected all aggregation steps, or if a timeout triggers.
>
>* Do you think it's worthwhile to dedicate a full thread per request to
>process its aggregations? In I/O bound cases, such as waiting for a WS
>reply, file download, etc. it feels wasteful to keep a thread parked for
>aggregations.
>
>* What do you think about calling the aggregation strategy from the
>thread(s) where the response(s) arrive(s)? (which could be from the
>multicast threadpool or a component thread e.g. TempReplyQueueManager in
>the case of JMS, depending on whether the component is sync or async).
>
>In essence, I'm considering a multicast v2 implementation (which would also
>affect RecipientList, Splitter, etc.) for 2.18, allowing the user to choose
>the implementation via DSL, e.g.
>
> .multicast().v2()
>
>It it's more efficient and feedback from users is good, we could make it
>the default multicast processor in Camel 3.0.
>
>Thoughts?
>
>Cheers,
>
>*Raúl Kripalani*
>PMC & Committer @ Apache Ignite, Apache Camel | Integration, Big Data and
>Messaging Engineer
>http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
>Blog: raul.io | twitter: @raulvk <https://twitter.com/raulvk>