You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Claus Ibsen <cl...@gmail.com> on 2009/04/30 07:32:49 UTC
Camel 2.0 Async Findings - Roadmap to a solution
Hi
You might remember my previous topic about the AsyncProcessor.
http://www.nabble.com/-DISCUSS---Camel-2.0---Internal-API-reworkings----Channel-and--AsyncProcessor-td23210093.html
I had some more time to thinker about it and I believe I have a
roadmap for a solution that will work nicely for all.
1) Internal house cleaning in camel-core
2) Fix as much of the current broken AsyncCallback
3) Introduce new async API in Camel 2.0
4) Mark the current AsyncCallback as @deprecated in Camel 2.0m2
Ad 1)
The AsyncProcessing internally in Camel is overused and it just makes
it much more complex. By doing a house cleaning the codebase is much
easier to work with and it works as before.
In fact it allows us to do fix the broken AsyncCallback also so we can
get notification when the async exchange is done.
Ad 2)
The current notification in Camel is in fact broken. You never get the
expected notification when the async exchange is done. The idea of
this API was to get 2 notifications
- the 1st is when the original synch thread is done (the boolean is
true), eg when you submit the job.
- and then the 2nd when the async thread is done (the boolean is
false) but this is broken in Camel and you will not receive it.
And of top of this there are 2 flaws in the current API
- the current code requires that you manaully add seda steps in the
route to turn the route from sync into async. There was no code that
did this "under the covers for you". However its doable and the in
fact its possible to route async. But an improve API would be better.
- The result of the async operation is not possible to get easily. No
you dont have access to the exchange that was done async. And no
exchange.getOut() will not give the result. What was missing in the
AsyncCallback API was to pass in the Exchange as well.
Ad 3)
I would like to introduce a new Async API that is more similar to the
JDK Future API. So you can use the Future get() to get hold of the
result.
This still needs some thinking and experimenting to see how its
doable. But an API that generally is like the Future would be much
better.
Ad 4)
I would also suggest that we (in Camel 2.0m2) mark AsyncCallback as
@deprecated with a note that we have planned a new Async API for Camel
2.0
Conclusion
========
I would like to go forward and get #1, (#2 and #4 would be great too)
committed into the codebase so we have a codebase in Camel that is
much easier to maintain and for new users to understand when they look
into the internals in Camel. And especially debugging and stepping
through the processing of Exchanges is much easier. As its just
regular method calls. And we get rid of the confusing code with what
to do in the AsyncCallbacks that was invading the internals in Camel.
Then I would start experimenting with the new API and report findings.
Feedback here is much welcome.
Any thoughts?
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
The issue is tracked at CAMEL-825
https://issues.apache.org/activemq/browse/CAMEL-825
I have just committed the implementation and updated the wiki pages.
Now both the seda and vm component work with request/reply, waiting
for the reply.
I added options to the endpoint, so you can control this behavior,
just like the threads DSL (part of the new Async API).
http://camel.apache.org/async.html
One feature we could consider is to add timeout support. Currently it
will wait until the route is comlete, regardless how long it takes.
On Fri, May 29, 2009 at 5:14 PM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> On Mon, May 25, 2009 at 9:28 AM, Claus Ibsen <cl...@gmail.com> wrote:
>> On Mon, May 25, 2009 at 9:00 AM, huntc <hu...@mac.com> wrote:
>>>
>>> Hi Claus,
>>>
>>> As per your blog request, I'd like to discuss the virtues of naming the
>>> async method "async" vs treating your 2.0 functionality as a new
>>> implementation of the existing "thread" method.
>
> Okay I got good tunes on iTunes so was in the mood to take a stab at this.
> Got the seda endpoint to support request/reply where it waits for the
> route to be complete before returning from the original caller.
>
> So you can do:
>
> from("direct:start").to("seda:a");
>
> from("seda:a").to("log:bar", "seda:b");
> from("seda:b").delay(10).to("direct:c");
>
> from("direct:c").transform(constant("Bye
> World")).to("mock:result");
>
> And the very first message we sent to direct:start that goes to seda:a
> will wait at seda:a until the message exchange is complete if its a
> request/reply message.
>
> So what we get is the "Bye World" as response.
>
> I will later add the same options as the *threads* DSL so you can
> overrule this behavior. Eg to not wait for a reply if you dont care.
>
>
> Christopher dont say we do not listen to the community. Thanks for the feedback.
>
>
>
>
>>>
>>> When I think about concurrency I think about multiple threads of execution -
>>> not whether something is asynchronous. You can have something being
>>> asynchronous without it being multi-threaded e.g. Javascript's
>>> XmlHttpRequest.
>>>
>>> Thread also implies just one thread. Perhaps renaming async to "threads" and
>>> deprecating "thread" may be the way to go? Specifying "threads" without a
>>> thread pool size should perhaps default to the number of processors + 1 as a
>>> rule... (as per MINA?).
>>
>> Good points. threads is a good name as well. And afterall the async
>> DSL is capable of waiting for a response so it behaves like a single
>> synchronous request/reply from the caller perspective. So using
>> threads could be a better name.
>>
>> I have added your idea of the defaults for number of threads in the
>> pool to the 2.x design page
>> http://cwiki.apache.org/confluence/display/CAMEL/Camel+2.0+Design
>>
>> We have improved thread pool configuration on the roadmap for 2.1.
>>
>>
>>
>>
>>>
>>> Thoughts?
>>>
>>> Kind regards,
>>> Christopher
>>> --
>>> View this message in context: http://www.nabble.com/Camel-2.0-Async-Findings---Roadmap-to-a-solution-tp23310165p23702159.html
>>> Sent from the Camel Development mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
On Mon, May 25, 2009 at 9:28 AM, Claus Ibsen <cl...@gmail.com> wrote:
> On Mon, May 25, 2009 at 9:00 AM, huntc <hu...@mac.com> wrote:
>>
>> Hi Claus,
>>
>> As per your blog request, I'd like to discuss the virtues of naming the
>> async method "async" vs treating your 2.0 functionality as a new
>> implementation of the existing "thread" method.
Okay I got good tunes on iTunes so was in the mood to take a stab at this.
Got the seda endpoint to support request/reply where it waits for the
route to be complete before returning from the original caller.
So you can do:
from("direct:start").to("seda:a");
from("seda:a").to("log:bar", "seda:b");
from("seda:b").delay(10).to("direct:c");
from("direct:c").transform(constant("Bye
World")).to("mock:result");
And the very first message we sent to direct:start that goes to seda:a
will wait at seda:a until the message exchange is complete if its a
request/reply message.
So what we get is the "Bye World" as response.
I will later add the same options as the *threads* DSL so you can
overrule this behavior. Eg to not wait for a reply if you dont care.
Christopher dont say we do not listen to the community. Thanks for the feedback.
>>
>> When I think about concurrency I think about multiple threads of execution -
>> not whether something is asynchronous. You can have something being
>> asynchronous without it being multi-threaded e.g. Javascript's
>> XmlHttpRequest.
>>
>> Thread also implies just one thread. Perhaps renaming async to "threads" and
>> deprecating "thread" may be the way to go? Specifying "threads" without a
>> thread pool size should perhaps default to the number of processors + 1 as a
>> rule... (as per MINA?).
>
> Good points. threads is a good name as well. And afterall the async
> DSL is capable of waiting for a response so it behaves like a single
> synchronous request/reply from the caller perspective. So using
> threads could be a better name.
>
> I have added your idea of the defaults for number of threads in the
> pool to the 2.x design page
> http://cwiki.apache.org/confluence/display/CAMEL/Camel+2.0+Design
>
> We have improved thread pool configuration on the roadmap for 2.1.
>
>
>
>
>>
>> Thoughts?
>>
>> Kind regards,
>> Christopher
>> --
>> View this message in context: http://www.nabble.com/Camel-2.0-Async-Findings---Roadmap-to-a-solution-tp23310165p23702159.html
>> Sent from the Camel Development mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
On Mon, May 25, 2009 at 9:00 AM, huntc <hu...@mac.com> wrote:
>
> Hi Claus,
>
> As per your blog request, I'd like to discuss the virtues of naming the
> async method "async" vs treating your 2.0 functionality as a new
> implementation of the existing "thread" method.
>
> When I think about concurrency I think about multiple threads of execution -
> not whether something is asynchronous. You can have something being
> asynchronous without it being multi-threaded e.g. Javascript's
> XmlHttpRequest.
>
> Thread also implies just one thread. Perhaps renaming async to "threads" and
> deprecating "thread" may be the way to go? Specifying "threads" without a
> thread pool size should perhaps default to the number of processors + 1 as a
> rule... (as per MINA?).
Good points. threads is a good name as well. And afterall the async
DSL is capable of waiting for a response so it behaves like a single
synchronous request/reply from the caller perspective. So using
threads could be a better name.
I have added your idea of the defaults for number of threads in the
pool to the 2.x design page
http://cwiki.apache.org/confluence/display/CAMEL/Camel+2.0+Design
We have improved thread pool configuration on the roadmap for 2.1.
>
> Thoughts?
>
> Kind regards,
> Christopher
> --
> View this message in context: http://www.nabble.com/Camel-2.0-Async-Findings---Roadmap-to-a-solution-tp23310165p23702159.html
> Sent from the Camel Development mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
On Mon, May 25, 2009 at 9:00 AM, huntc <hu...@mac.com> wrote:
>
> Hi Claus,
>
> As per your blog request, I'd like to discuss the virtues of naming the
> async method "async" vs treating your 2.0 functionality as a new
> implementation of the existing "thread" method.
>
> When I think about concurrency I think about multiple threads of execution -
> not whether something is asynchronous. You can have something being
> asynchronous without it being multi-threaded e.g. Javascript's
> XmlHttpRequest.
>
> Thread also implies just one thread. Perhaps renaming async to "threads" and
> deprecating "thread" may be the way to go? Specifying "threads" without a
> thread pool size should perhaps default to the number of processors + 1 as a
> rule... (as per MINA?).
Hi Christopher
Good points. I have renamed it from async to threads.
The default pool size is going to be part of an overall easier thread
managment and configuration of Camel planned for 2.1
>
> Thoughts?
>
> Kind regards,
> Christopher
> --
> View this message in context: http://www.nabble.com/Camel-2.0-Async-Findings---Roadmap-to-a-solution-tp23310165p23702159.html
> Sent from the Camel Development mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by huntc <hu...@mac.com>.
Hi Claus,
As per your blog request, I'd like to discuss the virtues of naming the
async method "async" vs treating your 2.0 functionality as a new
implementation of the existing "thread" method.
When I think about concurrency I think about multiple threads of execution -
not whether something is asynchronous. You can have something being
asynchronous without it being multi-threaded e.g. Javascript's
XmlHttpRequest.
Thread also implies just one thread. Perhaps renaming async to "threads" and
deprecating "thread" may be the way to go? Specifying "threads" without a
thread pool size should perhaps default to the number of processors + 1 as a
rule... (as per MINA?).
Thoughts?
Kind regards,
Christopher
--
View this message in context: http://www.nabble.com/Camel-2.0-Async-Findings---Roadmap-to-a-solution-tp23310165p23702159.html
Sent from the Camel Development mailing list archive at Nabble.com.
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, May 6, 2009 at 3:41 PM, Gert Vanthienen
<ge...@gmail.com> wrote:
> Hi Claus,
>
> Nice work on cleaning up the async API for Camel! Using well-known
> java.util.concurrency classes to build the API is a good idea, makes
> it a lot more comprehensible for people. Just a few questions that
> come to mind...
> - How does this work relate to introducing the Channel API? Will we
> have a means for using async channels in the route transparently or
> are the two unrelated?
Per see the two are unrelated as such. But in the future we can much
more easily introduce async channels.
If you want for instance each processing step to be done async step by step :)
But currently the Channel we have is not so flexible but the ground
work has been done for the future work.
> - What happens with the original thread after the async()? I'm
> guessing it will wait for the async work to be done before continuing,
> right?
Note the async DSL I did in the patch was an experiment but will bring
it back in to replace the old Thread DSL.
Well the original thread gets the Future<Exchange> handle as a result.
So it basically returns this back to the original caller.
Then the original caller can do other work, and then retrieve the
async thread result when he/she likes with the future handle get
method.
> - Do all the threads come from a single thread pool? Do we have a
> means to configure that pool? I guess my main question is, how likely
> are we to deadlock the entire Route by having all the threads either
> waiting on some Future or waiting to get another thread from the pool?
Good question. I am also aligning the configuration of thread pools in
Camel 2.0. So its based on ExecutorService and have the same API /
Spring attribute.
This allows end users to define thread pools as they like more easily
and configure it.
And at the same time we should provide some default pools and pool
sizes. Some uses 5 and others 10 in the current code base. But
basically each processor creates its own thread pool if none
configured, and they default to a size of either 5 or 10.
When I get a bit more work done lets re asset and find a good solution
for thread pool management in Camel 2.0
For instance some users with J2EE servers would like to define that
Camel should use threads provided from this server.
>
> Just wondering if we could somehow put an Erlang-style message-passing
> concurrency mechanism underneath our Route afterwards -- in a
> transparent way so people wouldn't have to worry about this.
I guess we have enough work just to keep the Scala DSL up to date. But
yeah concurrency and parallel work is challeging the way we program
today in Java.
Bug be careful James might read this and ask you to do some prototyping :)
>
> Regards,
>
> Gert Vanthienen
> ------------------------
> Open Source SOA: http://fusesource.com
> Blog: http://gertvanthienen.blogspot.com/
>
>
>
> 2009/5/6 Claus Ibsen <cl...@gmail.com>:
>> Hi
>>
>> Status update
>>
>> On Wed, May 6, 2009 at 9:17 AM, Claus Ibsen <cl...@gmail.com> wrote:
>>> Hi
>>>
>>> I have committed first cut of the new Async API to Camel trunk.
>>> See ticket CAMEL-1572 for svn revision and details.
>>>
>>> The remaining work:
>>> - Migrate MulticastProcessor
>> DONE
>>
>>> - Remove last piece of old API classes (2 interfaces)
>> DONE
>>
>>> - Introduce Async DSL to replace Thread DSL and with clear intention
>>> of turning route into async mode and support thread pools using JDK
>>> executor service.
>>> - Support Jetty continuation with async
>>>
>>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Interview with me:
http://architects.dzone.com/articles/interview-claus-ibsen-about?mz=7893-progress
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, May 12, 2009 at 10:25 AM, Willem Jiang <wi...@gmail.com> wrote:
> Hi Claus,
>
> Thanks for your explanation.
> Could you add the road map of call back support into the Camel 2.0
> Design page[1]?
>
> [1] http://camel.apache.org/camel-20-design.html
Two minds think alike. I have just updated the design page and moved
the UnitOfWork callbacks from 2.1 to 2.0.
>
> Willem
>
> Claus Ibsen wrote:
>> On Mon, May 11, 2009 at 9:32 AM, Willem Jiang <wi...@gmail.com> wrote:
>>> Hi Claus
>>>
>>> I really enjoy to read the document that you write.
>>> You give me a good explanation and compare of Camel Sync and Async
>>> processing API.
>>>
>>> Now I just have quick question for the replacement of old AsyncCallback
>>> API Done().
>>> Let's take the Asynchronous Request Only message sending as an example.
>>>
>>> The user could want to know if the request is processed succeed, but he
>>> doesn't want to his sending client checks every Feature object for
>>> return message, he just want to register a simple call back method for
>>> doing some revert work if the message can't be processed rightly.
>>>
>>> How can the new Async API help us to implement this user story.
>> The tip kinda explains that:
>>
>> In case you want to know whether the Async Request Only failed, then
>> you can use the Future handle and invoke get() and if it throws a
>> ExecutionException then the processing failed. The caused exception is
>> wrapped. You can invoke isDone() first to test whether the task is
>> done or still in progress. Otherwise invoking get() will wait until
>> the task is done.
>>
>>
>> So if you want to check whether an exception occurred or not you can do
>>
>> public boolean didTheJobSucceed(Future future) {
>> try {
>> future.get();
>> return true;
>> } catch (ExecutionException e) {
>> return false;
>> }
>> }
>>
>> But its not 100% what you want as you want a callback that is invoked
>> when the job is done.
>> Well the JDK concurrency API does not support that easily. You can not
>> register custom callbacks, that are automatic invoked when a task is
>> complete.
>>
>>
>> But dont dispair. In Camel we have the UnitOfWork for that. The idea
>> is that its where end users and camel components can register
>> callbacks
>> when a Camel Exchange is complete or failed.
>>
>> We have planned for Camel 2.1 to give this an overhaul as well, so we
>> got nice syntax sugar in the DSL itself so you can easily add custom
>> callbacks, that can be camel routes as well. Eg to easily send an
>> email in case of a route failure.
>>
>> In the mean time you can use the UnitOfWork and add a callback
>> yourself from a processor.
>> There is a addSynchronization method on the UoW
>>
>> exchange.getUnitOfWork().addSynchronization(mySync)
>>
>>
>> So Willem we will get this improved in the future as well. You can do
>> it now but will be easier and nicer in Camel 2.1.
>> Or if we find the time then we can maybe make it for 2.0 as well. I
>> guess the main problem before holding it back was the old Async API
>> that just got to complex and broken. So I do envision it should be not
>> that hard to do now.
>>
>>
>>> BTW, Since Spring store the transaction information in the thread local
>>> variable , we need to make sure the rollback method is called in the
>>> same thread, so this is common user story in camel.
>>
>> Yeah for transactions you should not mix threads. This problem can
>> occur if you use the async() DSL as it will submit a new task that are
>> executed in a new thread. But for the client API using the
>> ProducerTemplate there are no problem, as the transaction should
>> usually not span both the client + "camel".
>>
>> But for sure that is something you should consider when doing transactions.
>>
>> Everyting has its pros/cons. Also async messaging.
>>
>>
>>
>>> Willem
>>>
>>> Claus Ibsen wrote:
>>>> On Fri, May 8, 2009 at 8:52 AM, Claus Ibsen <cl...@gmail.com> wrote:
>>>>> On Wed, May 6, 2009 at 3:41 PM, Gert Vanthienen
>>>>> <ge...@gmail.com> wrote:
>>>>>> Hi Claus,
>>>>>>
>>>>>> Nice work on cleaning up the async API for Camel! Using well-known
>>>>>> java.util.concurrency classes to build the API is a good idea, makes
>>>>>> it a lot more comprehensible for people. Just a few questions that
>>>>>> come to mind...
>>>>>> - How does this work relate to introducing the Channel API? Will we
>>>>>> have a means for using async channels in the route transparently or
>>>>>> are the two unrelated?
>>>>>> - What happens with the original thread after the async()? I'm
>>>>>> guessing it will wait for the async work to be done before continuing,
>>>>>> right?
>>>>> I have just committed some improvements to the async DSL.
>>>>> There is an option you can configure: waitForTaskToComplete.
>>>>>
>>>>> This option is an enum accepting 3 options
>>>>> - Always
>>>>> - Newer
>>>>> - IfReplyExpected
>>>>>
>>>>> The first two options is self describing.
>>>>> The last one will wait if the exchange is OUT capable, otherwise not.
>>>>>
>>>>> The current default is Always,
>>>> I have given it some more thought and changed the default to
>>>> IfReplyExpected, as its kinda what eg. JMS Mina and other components
>>>> also does already.
>>>>
>>>> BTW: I have created a new wiki page for all this new Async API. Anyone
>>>> care to read and review is much appreciated.
>>>> http://camel.apache.org/async
>>>>
>>>>
>>>>> However I am wondering if the default should be IfReplyExpected instead?
>>>>> Then it works as JMS, Mina etc. that also only wait if a reply is expected.
>>>>>
>>>>> What do people think?
>>>>>
>>>>> Okay what is the difference between Always and IfReplyExpected then?
>>>>> If for instance we send an InOut message then there are no difference,
>>>>> they both wait.
>>>>>
>>>>> However if we send an InOnly then only Always will wait. So what can
>>>>> that be used for?
>>>>> Well in case the async task failed with an exception, and since we
>>>>> wait, we will be notified
>>>>> of this exception and have a chance to react then. That is the difference.
>>>>>
>>>>> Any thoughts?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> - Do all the threads come from a single thread pool? Do we have a
>>>>>> means to configure that pool? I guess my main question is, how likely
>>>>>> are we to deadlock the entire Route by having all the threads either
>>>>>> waiting on some Future or waiting to get another thread from the pool?
>>>>>>
>>>>>> Just wondering if we could somehow put an Erlang-style message-passing
>>>>>> concurrency mechanism underneath our Route afterwards -- in a
>>>>>> transparent way so people wouldn't have to worry about this.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Gert Vanthienen
>>>>>> ------------------------
>>>>>> Open Source SOA: http://fusesource.com
>>>>>> Blog: http://gertvanthienen.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2009/5/6 Claus Ibsen <cl...@gmail.com>:
>>>>>>> Hi
>>>>>>>
>>>>>>> Status update
>>>>>>>
>>>>>>> On Wed, May 6, 2009 at 9:17 AM, Claus Ibsen <cl...@gmail.com> wrote:
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> I have committed first cut of the new Async API to Camel trunk.
>>>>>>>> See ticket CAMEL-1572 for svn revision and details.
>>>>>>>>
>>>>>>>> The remaining work:
>>>>>>>> - Migrate MulticastProcessor
>>>>>>> DONE
>>>>>>>
>>>>>>>> - Remove last piece of old API classes (2 interfaces)
>>>>>>> DONE
>>>>>>>
>>>>>>>> - Introduce Async DSL to replace Thread DSL and with clear intention
>>>>>>>> of turning route into async mode and support thread pools using JDK
>>>>>>>> executor service.
>>>>>>>> - Support Jetty continuation with async
>>>>>>>>
>>>>>
>>>>> --
>>>>> Claus Ibsen
>>>>> Apache Camel Committer
>>>>>
>>>>> Open Source Integration: http://fusesource.com
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Twitter: http://twitter.com/davsclaus
>>>>> Apache Camel Reference Card:
>>>>> http://refcardz.dzone.com/refcardz/enterprise-integration
>>>>> Interview with me:
>>>>> http://architects.dzone.com/articles/interview-claus-ibsen-about?mz=7893-progress
>>>>>
>>>>
>>>>
>>>
>>
>>
>>
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Interview with me:
http://architects.dzone.com/articles/interview-claus-ibsen-about?mz=7893-progress
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Willem Jiang <wi...@gmail.com>.
Hi Claus,
Thanks for your explanation.
Could you add the road map of call back support into the Camel 2.0
Design page[1]?
[1] http://camel.apache.org/camel-20-design.html
Willem
Claus Ibsen wrote:
> On Mon, May 11, 2009 at 9:32 AM, Willem Jiang <wi...@gmail.com> wrote:
>> Hi Claus
>>
>> I really enjoy to read the document that you write.
>> You give me a good explanation and compare of Camel Sync and Async
>> processing API.
>>
>> Now I just have quick question for the replacement of old AsyncCallback
>> API Done().
>> Let's take the Asynchronous Request Only message sending as an example.
>>
>> The user could want to know if the request is processed succeed, but he
>> doesn't want to his sending client checks every Feature object for
>> return message, he just want to register a simple call back method for
>> doing some revert work if the message can't be processed rightly.
>>
>> How can the new Async API help us to implement this user story.
> The tip kinda explains that:
>
> In case you want to know whether the Async Request Only failed, then
> you can use the Future handle and invoke get() and if it throws a
> ExecutionException then the processing failed. The caused exception is
> wrapped. You can invoke isDone() first to test whether the task is
> done or still in progress. Otherwise invoking get() will wait until
> the task is done.
>
>
> So if you want to check whether an exception occurred or not you can do
>
> public boolean didTheJobSucceed(Future future) {
> try {
> future.get();
> return true;
> } catch (ExecutionException e) {
> return false;
> }
> }
>
> But its not 100% what you want as you want a callback that is invoked
> when the job is done.
> Well the JDK concurrency API does not support that easily. You can not
> register custom callbacks, that are automatic invoked when a task is
> complete.
>
>
> But dont dispair. In Camel we have the UnitOfWork for that. The idea
> is that its where end users and camel components can register
> callbacks
> when a Camel Exchange is complete or failed.
>
> We have planned for Camel 2.1 to give this an overhaul as well, so we
> got nice syntax sugar in the DSL itself so you can easily add custom
> callbacks, that can be camel routes as well. Eg to easily send an
> email in case of a route failure.
>
> In the mean time you can use the UnitOfWork and add a callback
> yourself from a processor.
> There is a addSynchronization method on the UoW
>
> exchange.getUnitOfWork().addSynchronization(mySync)
>
>
> So Willem we will get this improved in the future as well. You can do
> it now but will be easier and nicer in Camel 2.1.
> Or if we find the time then we can maybe make it for 2.0 as well. I
> guess the main problem before holding it back was the old Async API
> that just got to complex and broken. So I do envision it should be not
> that hard to do now.
>
>
>> BTW, Since Spring store the transaction information in the thread local
>> variable , we need to make sure the rollback method is called in the
>> same thread, so this is common user story in camel.
>
> Yeah for transactions you should not mix threads. This problem can
> occur if you use the async() DSL as it will submit a new task that are
> executed in a new thread. But for the client API using the
> ProducerTemplate there are no problem, as the transaction should
> usually not span both the client + "camel".
>
> But for sure that is something you should consider when doing transactions.
>
> Everyting has its pros/cons. Also async messaging.
>
>
>
>> Willem
>>
>> Claus Ibsen wrote:
>>> On Fri, May 8, 2009 at 8:52 AM, Claus Ibsen <cl...@gmail.com> wrote:
>>>> On Wed, May 6, 2009 at 3:41 PM, Gert Vanthienen
>>>> <ge...@gmail.com> wrote:
>>>>> Hi Claus,
>>>>>
>>>>> Nice work on cleaning up the async API for Camel! Using well-known
>>>>> java.util.concurrency classes to build the API is a good idea, makes
>>>>> it a lot more comprehensible for people. Just a few questions that
>>>>> come to mind...
>>>>> - How does this work relate to introducing the Channel API? Will we
>>>>> have a means for using async channels in the route transparently or
>>>>> are the two unrelated?
>>>>> - What happens with the original thread after the async()? I'm
>>>>> guessing it will wait for the async work to be done before continuing,
>>>>> right?
>>>> I have just committed some improvements to the async DSL.
>>>> There is an option you can configure: waitForTaskToComplete.
>>>>
>>>> This option is an enum accepting 3 options
>>>> - Always
>>>> - Newer
>>>> - IfReplyExpected
>>>>
>>>> The first two options is self describing.
>>>> The last one will wait if the exchange is OUT capable, otherwise not.
>>>>
>>>> The current default is Always,
>>> I have given it some more thought and changed the default to
>>> IfReplyExpected, as its kinda what eg. JMS Mina and other components
>>> also does already.
>>>
>>> BTW: I have created a new wiki page for all this new Async API. Anyone
>>> care to read and review is much appreciated.
>>> http://camel.apache.org/async
>>>
>>>
>>>> However I am wondering if the default should be IfReplyExpected instead?
>>>> Then it works as JMS, Mina etc. that also only wait if a reply is expected.
>>>>
>>>> What do people think?
>>>>
>>>> Okay what is the difference between Always and IfReplyExpected then?
>>>> If for instance we send an InOut message then there are no difference,
>>>> they both wait.
>>>>
>>>> However if we send an InOnly then only Always will wait. So what can
>>>> that be used for?
>>>> Well in case the async task failed with an exception, and since we
>>>> wait, we will be notified
>>>> of this exception and have a chance to react then. That is the difference.
>>>>
>>>> Any thoughts?
>>>>
>>>>
>>>>
>>>>
>>>>> - Do all the threads come from a single thread pool? Do we have a
>>>>> means to configure that pool? I guess my main question is, how likely
>>>>> are we to deadlock the entire Route by having all the threads either
>>>>> waiting on some Future or waiting to get another thread from the pool?
>>>>>
>>>>> Just wondering if we could somehow put an Erlang-style message-passing
>>>>> concurrency mechanism underneath our Route afterwards -- in a
>>>>> transparent way so people wouldn't have to worry about this.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Gert Vanthienen
>>>>> ------------------------
>>>>> Open Source SOA: http://fusesource.com
>>>>> Blog: http://gertvanthienen.blogspot.com/
>>>>>
>>>>>
>>>>>
>>>>> 2009/5/6 Claus Ibsen <cl...@gmail.com>:
>>>>>> Hi
>>>>>>
>>>>>> Status update
>>>>>>
>>>>>> On Wed, May 6, 2009 at 9:17 AM, Claus Ibsen <cl...@gmail.com> wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>> I have committed first cut of the new Async API to Camel trunk.
>>>>>>> See ticket CAMEL-1572 for svn revision and details.
>>>>>>>
>>>>>>> The remaining work:
>>>>>>> - Migrate MulticastProcessor
>>>>>> DONE
>>>>>>
>>>>>>> - Remove last piece of old API classes (2 interfaces)
>>>>>> DONE
>>>>>>
>>>>>>> - Introduce Async DSL to replace Thread DSL and with clear intention
>>>>>>> of turning route into async mode and support thread pools using JDK
>>>>>>> executor service.
>>>>>>> - Support Jetty continuation with async
>>>>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>> Apache Camel Reference Card:
>>>> http://refcardz.dzone.com/refcardz/enterprise-integration
>>>> Interview with me:
>>>> http://architects.dzone.com/articles/interview-claus-ibsen-about?mz=7893-progress
>>>>
>>>
>>>
>>
>
>
>
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
On Mon, May 11, 2009 at 9:32 AM, Willem Jiang <wi...@gmail.com> wrote:
> Hi Claus
>
> I really enjoy to read the document that you write.
> You give me a good explanation and compare of Camel Sync and Async
> processing API.
>
> Now I just have quick question for the replacement of old AsyncCallback
> API Done().
> Let's take the Asynchronous Request Only message sending as an example.
>
> The user could want to know if the request is processed succeed, but he
> doesn't want to his sending client checks every Feature object for
> return message, he just want to register a simple call back method for
> doing some revert work if the message can't be processed rightly.
>
> How can the new Async API help us to implement this user story.
The tip kinda explains that:
In case you want to know whether the Async Request Only failed, then
you can use the Future handle and invoke get() and if it throws a
ExecutionException then the processing failed. The caused exception is
wrapped. You can invoke isDone() first to test whether the task is
done or still in progress. Otherwise invoking get() will wait until
the task is done.
So if you want to check whether an exception occurred or not you can do
public boolean didTheJobSucceed(Future future) {
try {
future.get();
return true;
} catch (ExecutionException e) {
return false;
}
}
But its not 100% what you want as you want a callback that is invoked
when the job is done.
Well the JDK concurrency API does not support that easily. You can not
register custom callbacks, that are automatic invoked when a task is
complete.
But dont dispair. In Camel we have the UnitOfWork for that. The idea
is that its where end users and camel components can register
callbacks
when a Camel Exchange is complete or failed.
We have planned for Camel 2.1 to give this an overhaul as well, so we
got nice syntax sugar in the DSL itself so you can easily add custom
callbacks, that can be camel routes as well. Eg to easily send an
email in case of a route failure.
In the mean time you can use the UnitOfWork and add a callback
yourself from a processor.
There is a addSynchronization method on the UoW
exchange.getUnitOfWork().addSynchronization(mySync)
So Willem we will get this improved in the future as well. You can do
it now but will be easier and nicer in Camel 2.1.
Or if we find the time then we can maybe make it for 2.0 as well. I
guess the main problem before holding it back was the old Async API
that just got to complex and broken. So I do envision it should be not
that hard to do now.
>
> BTW, Since Spring store the transaction information in the thread local
> variable , we need to make sure the rollback method is called in the
> same thread, so this is common user story in camel.
Yeah for transactions you should not mix threads. This problem can
occur if you use the async() DSL as it will submit a new task that are
executed in a new thread. But for the client API using the
ProducerTemplate there are no problem, as the transaction should
usually not span both the client + "camel".
But for sure that is something you should consider when doing transactions.
Everyting has its pros/cons. Also async messaging.
>
> Willem
>
> Claus Ibsen wrote:
>> On Fri, May 8, 2009 at 8:52 AM, Claus Ibsen <cl...@gmail.com> wrote:
>>> On Wed, May 6, 2009 at 3:41 PM, Gert Vanthienen
>>> <ge...@gmail.com> wrote:
>>>> Hi Claus,
>>>>
>>>> Nice work on cleaning up the async API for Camel! Using well-known
>>>> java.util.concurrency classes to build the API is a good idea, makes
>>>> it a lot more comprehensible for people. Just a few questions that
>>>> come to mind...
>>>> - How does this work relate to introducing the Channel API? Will we
>>>> have a means for using async channels in the route transparently or
>>>> are the two unrelated?
>>>> - What happens with the original thread after the async()? I'm
>>>> guessing it will wait for the async work to be done before continuing,
>>>> right?
>>> I have just committed some improvements to the async DSL.
>>> There is an option you can configure: waitForTaskToComplete.
>>>
>>> This option is an enum accepting 3 options
>>> - Always
>>> - Newer
>>> - IfReplyExpected
>>>
>>> The first two options is self describing.
>>> The last one will wait if the exchange is OUT capable, otherwise not.
>>>
>>> The current default is Always,
>> I have given it some more thought and changed the default to
>> IfReplyExpected, as its kinda what eg. JMS Mina and other components
>> also does already.
>>
>> BTW: I have created a new wiki page for all this new Async API. Anyone
>> care to read and review is much appreciated.
>> http://camel.apache.org/async
>>
>>
>>> However I am wondering if the default should be IfReplyExpected instead?
>>> Then it works as JMS, Mina etc. that also only wait if a reply is expected.
>>>
>>> What do people think?
>>>
>>> Okay what is the difference between Always and IfReplyExpected then?
>>> If for instance we send an InOut message then there are no difference,
>>> they both wait.
>>>
>>> However if we send an InOnly then only Always will wait. So what can
>>> that be used for?
>>> Well in case the async task failed with an exception, and since we
>>> wait, we will be notified
>>> of this exception and have a chance to react then. That is the difference.
>>>
>>> Any thoughts?
>>>
>>>
>>>
>>>
>>>> - Do all the threads come from a single thread pool? Do we have a
>>>> means to configure that pool? I guess my main question is, how likely
>>>> are we to deadlock the entire Route by having all the threads either
>>>> waiting on some Future or waiting to get another thread from the pool?
>>>>
>>>> Just wondering if we could somehow put an Erlang-style message-passing
>>>> concurrency mechanism underneath our Route afterwards -- in a
>>>> transparent way so people wouldn't have to worry about this.
>>>>
>>>> Regards,
>>>>
>>>> Gert Vanthienen
>>>> ------------------------
>>>> Open Source SOA: http://fusesource.com
>>>> Blog: http://gertvanthienen.blogspot.com/
>>>>
>>>>
>>>>
>>>> 2009/5/6 Claus Ibsen <cl...@gmail.com>:
>>>>> Hi
>>>>>
>>>>> Status update
>>>>>
>>>>> On Wed, May 6, 2009 at 9:17 AM, Claus Ibsen <cl...@gmail.com> wrote:
>>>>>> Hi
>>>>>>
>>>>>> I have committed first cut of the new Async API to Camel trunk.
>>>>>> See ticket CAMEL-1572 for svn revision and details.
>>>>>>
>>>>>> The remaining work:
>>>>>> - Migrate MulticastProcessor
>>>>> DONE
>>>>>
>>>>>> - Remove last piece of old API classes (2 interfaces)
>>>>> DONE
>>>>>
>>>>>> - Introduce Async DSL to replace Thread DSL and with clear intention
>>>>>> of turning route into async mode and support thread pools using JDK
>>>>>> executor service.
>>>>>> - Support Jetty continuation with async
>>>>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>> Apache Camel Reference Card:
>>> http://refcardz.dzone.com/refcardz/enterprise-integration
>>> Interview with me:
>>> http://architects.dzone.com/articles/interview-claus-ibsen-about?mz=7893-progress
>>>
>>
>>
>>
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Interview with me:
http://architects.dzone.com/articles/interview-claus-ibsen-about?mz=7893-progress
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Willem Jiang <wi...@gmail.com>.
Hi Claus
I really enjoy to read the document that you write.
You give me a good explanation and compare of Camel Sync and Async
processing API.
Now I just have quick question for the replacement of old AsyncCallback
API Done().
Let's take the Asynchronous Request Only message sending as an example.
The user could want to know if the request is processed succeed, but he
doesn't want to his sending client checks every Feature object for
return message, he just want to register a simple call back method for
doing some revert work if the message can't be processed rightly.
How can the new Async API help us to implement this user story.
BTW, Since Spring store the transaction information in the thread local
variable , we need to make sure the rollback method is called in the
same thread, so this is common user story in camel.
Willem
Claus Ibsen wrote:
> On Fri, May 8, 2009 at 8:52 AM, Claus Ibsen <cl...@gmail.com> wrote:
>> On Wed, May 6, 2009 at 3:41 PM, Gert Vanthienen
>> <ge...@gmail.com> wrote:
>>> Hi Claus,
>>>
>>> Nice work on cleaning up the async API for Camel! Using well-known
>>> java.util.concurrency classes to build the API is a good idea, makes
>>> it a lot more comprehensible for people. Just a few questions that
>>> come to mind...
>>> - How does this work relate to introducing the Channel API? Will we
>>> have a means for using async channels in the route transparently or
>>> are the two unrelated?
>>> - What happens with the original thread after the async()? I'm
>>> guessing it will wait for the async work to be done before continuing,
>>> right?
>> I have just committed some improvements to the async DSL.
>> There is an option you can configure: waitForTaskToComplete.
>>
>> This option is an enum accepting 3 options
>> - Always
>> - Newer
>> - IfReplyExpected
>>
>> The first two options is self describing.
>> The last one will wait if the exchange is OUT capable, otherwise not.
>>
>> The current default is Always,
> I have given it some more thought and changed the default to
> IfReplyExpected, as its kinda what eg. JMS Mina and other components
> also does already.
>
> BTW: I have created a new wiki page for all this new Async API. Anyone
> care to read and review is much appreciated.
> http://camel.apache.org/async
>
>
>> However I am wondering if the default should be IfReplyExpected instead?
>> Then it works as JMS, Mina etc. that also only wait if a reply is expected.
>>
>> What do people think?
>>
>> Okay what is the difference between Always and IfReplyExpected then?
>> If for instance we send an InOut message then there are no difference,
>> they both wait.
>>
>> However if we send an InOnly then only Always will wait. So what can
>> that be used for?
>> Well in case the async task failed with an exception, and since we
>> wait, we will be notified
>> of this exception and have a chance to react then. That is the difference.
>>
>> Any thoughts?
>>
>>
>>
>>
>>> - Do all the threads come from a single thread pool? Do we have a
>>> means to configure that pool? I guess my main question is, how likely
>>> are we to deadlock the entire Route by having all the threads either
>>> waiting on some Future or waiting to get another thread from the pool?
>>>
>>> Just wondering if we could somehow put an Erlang-style message-passing
>>> concurrency mechanism underneath our Route afterwards -- in a
>>> transparent way so people wouldn't have to worry about this.
>>>
>>> Regards,
>>>
>>> Gert Vanthienen
>>> ------------------------
>>> Open Source SOA: http://fusesource.com
>>> Blog: http://gertvanthienen.blogspot.com/
>>>
>>>
>>>
>>> 2009/5/6 Claus Ibsen <cl...@gmail.com>:
>>>> Hi
>>>>
>>>> Status update
>>>>
>>>> On Wed, May 6, 2009 at 9:17 AM, Claus Ibsen <cl...@gmail.com> wrote:
>>>>> Hi
>>>>>
>>>>> I have committed first cut of the new Async API to Camel trunk.
>>>>> See ticket CAMEL-1572 for svn revision and details.
>>>>>
>>>>> The remaining work:
>>>>> - Migrate MulticastProcessor
>>>> DONE
>>>>
>>>>> - Remove last piece of old API classes (2 interfaces)
>>>> DONE
>>>>
>>>>> - Introduce Async DSL to replace Thread DSL and with clear intention
>>>>> of turning route into async mode and support thread pools using JDK
>>>>> executor service.
>>>>> - Support Jetty continuation with async
>>>>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>> Apache Camel Reference Card:
>> http://refcardz.dzone.com/refcardz/enterprise-integration
>> Interview with me:
>> http://architects.dzone.com/articles/interview-claus-ibsen-about?mz=7893-progress
>>
>
>
>
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, May 8, 2009 at 8:52 AM, Claus Ibsen <cl...@gmail.com> wrote:
> On Wed, May 6, 2009 at 3:41 PM, Gert Vanthienen
> <ge...@gmail.com> wrote:
>> Hi Claus,
>>
>> Nice work on cleaning up the async API for Camel! Using well-known
>> java.util.concurrency classes to build the API is a good idea, makes
>> it a lot more comprehensible for people. Just a few questions that
>> come to mind...
>> - How does this work relate to introducing the Channel API? Will we
>> have a means for using async channels in the route transparently or
>> are the two unrelated?
>
>> - What happens with the original thread after the async()? I'm
>> guessing it will wait for the async work to be done before continuing,
>> right?
>
> I have just committed some improvements to the async DSL.
> There is an option you can configure: waitForTaskToComplete.
>
> This option is an enum accepting 3 options
> - Always
> - Newer
> - IfReplyExpected
>
> The first two options is self describing.
> The last one will wait if the exchange is OUT capable, otherwise not.
>
> The current default is Always,
I have given it some more thought and changed the default to
IfReplyExpected, as its kinda what eg. JMS Mina and other components
also does already.
BTW: I have created a new wiki page for all this new Async API. Anyone
care to read and review is much appreciated.
http://camel.apache.org/async
>
> However I am wondering if the default should be IfReplyExpected instead?
> Then it works as JMS, Mina etc. that also only wait if a reply is expected.
>
> What do people think?
>
> Okay what is the difference between Always and IfReplyExpected then?
> If for instance we send an InOut message then there are no difference,
> they both wait.
>
> However if we send an InOnly then only Always will wait. So what can
> that be used for?
> Well in case the async task failed with an exception, and since we
> wait, we will be notified
> of this exception and have a chance to react then. That is the difference.
>
> Any thoughts?
>
>
>
>
>> - Do all the threads come from a single thread pool? Do we have a
>> means to configure that pool? I guess my main question is, how likely
>> are we to deadlock the entire Route by having all the threads either
>> waiting on some Future or waiting to get another thread from the pool?
>>
>> Just wondering if we could somehow put an Erlang-style message-passing
>> concurrency mechanism underneath our Route afterwards -- in a
>> transparent way so people wouldn't have to worry about this.
>>
>> Regards,
>>
>> Gert Vanthienen
>> ------------------------
>> Open Source SOA: http://fusesource.com
>> Blog: http://gertvanthienen.blogspot.com/
>>
>>
>>
>> 2009/5/6 Claus Ibsen <cl...@gmail.com>:
>>> Hi
>>>
>>> Status update
>>>
>>> On Wed, May 6, 2009 at 9:17 AM, Claus Ibsen <cl...@gmail.com> wrote:
>>>> Hi
>>>>
>>>> I have committed first cut of the new Async API to Camel trunk.
>>>> See ticket CAMEL-1572 for svn revision and details.
>>>>
>>>> The remaining work:
>>>> - Migrate MulticastProcessor
>>> DONE
>>>
>>>> - Remove last piece of old API classes (2 interfaces)
>>> DONE
>>>
>>>> - Introduce Async DSL to replace Thread DSL and with clear intention
>>>> of turning route into async mode and support thread pools using JDK
>>>> executor service.
>>>> - Support Jetty continuation with async
>>>>
>>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
> Apache Camel Reference Card:
> http://refcardz.dzone.com/refcardz/enterprise-integration
> Interview with me:
> http://architects.dzone.com/articles/interview-claus-ibsen-about?mz=7893-progress
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Interview with me:
http://architects.dzone.com/articles/interview-claus-ibsen-about?mz=7893-progress
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, May 6, 2009 at 3:41 PM, Gert Vanthienen
<ge...@gmail.com> wrote:
> Hi Claus,
>
> Nice work on cleaning up the async API for Camel! Using well-known
> java.util.concurrency classes to build the API is a good idea, makes
> it a lot more comprehensible for people. Just a few questions that
> come to mind...
> - How does this work relate to introducing the Channel API? Will we
> have a means for using async channels in the route transparently or
> are the two unrelated?
> - What happens with the original thread after the async()? I'm
> guessing it will wait for the async work to be done before continuing,
> right?
I have just committed some improvements to the async DSL.
There is an option you can configure: waitForTaskToComplete.
This option is an enum accepting 3 options
- Always
- Newer
- IfReplyExpected
The first two options is self describing.
The last one will wait if the exchange is OUT capable, otherwise not.
The current default is Always,
However I am wondering if the default should be IfReplyExpected instead?
Then it works as JMS, Mina etc. that also only wait if a reply is expected.
What do people think?
Okay what is the difference between Always and IfReplyExpected then?
If for instance we send an InOut message then there are no difference,
they both wait.
However if we send an InOnly then only Always will wait. So what can
that be used for?
Well in case the async task failed with an exception, and since we
wait, we will be notified
of this exception and have a chance to react then. That is the difference.
Any thoughts?
> - Do all the threads come from a single thread pool? Do we have a
> means to configure that pool? I guess my main question is, how likely
> are we to deadlock the entire Route by having all the threads either
> waiting on some Future or waiting to get another thread from the pool?
>
> Just wondering if we could somehow put an Erlang-style message-passing
> concurrency mechanism underneath our Route afterwards -- in a
> transparent way so people wouldn't have to worry about this.
>
> Regards,
>
> Gert Vanthienen
> ------------------------
> Open Source SOA: http://fusesource.com
> Blog: http://gertvanthienen.blogspot.com/
>
>
>
> 2009/5/6 Claus Ibsen <cl...@gmail.com>:
>> Hi
>>
>> Status update
>>
>> On Wed, May 6, 2009 at 9:17 AM, Claus Ibsen <cl...@gmail.com> wrote:
>>> Hi
>>>
>>> I have committed first cut of the new Async API to Camel trunk.
>>> See ticket CAMEL-1572 for svn revision and details.
>>>
>>> The remaining work:
>>> - Migrate MulticastProcessor
>> DONE
>>
>>> - Remove last piece of old API classes (2 interfaces)
>> DONE
>>
>>> - Introduce Async DSL to replace Thread DSL and with clear intention
>>> of turning route into async mode and support thread pools using JDK
>>> executor service.
>>> - Support Jetty continuation with async
>>>
>>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Interview with me:
http://architects.dzone.com/articles/interview-claus-ibsen-about?mz=7893-progress
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Gert Vanthienen <ge...@gmail.com>.
Hi Claus,
Nice work on cleaning up the async API for Camel! Using well-known
java.util.concurrency classes to build the API is a good idea, makes
it a lot more comprehensible for people. Just a few questions that
come to mind...
- How does this work relate to introducing the Channel API? Will we
have a means for using async channels in the route transparently or
are the two unrelated?
- What happens with the original thread after the async()? I'm
guessing it will wait for the async work to be done before continuing,
right?
- Do all the threads come from a single thread pool? Do we have a
means to configure that pool? I guess my main question is, how likely
are we to deadlock the entire Route by having all the threads either
waiting on some Future or waiting to get another thread from the pool?
Just wondering if we could somehow put an Erlang-style message-passing
concurrency mechanism underneath our Route afterwards -- in a
transparent way so people wouldn't have to worry about this.
Regards,
Gert Vanthienen
------------------------
Open Source SOA: http://fusesource.com
Blog: http://gertvanthienen.blogspot.com/
2009/5/6 Claus Ibsen <cl...@gmail.com>:
> Hi
>
> Status update
>
> On Wed, May 6, 2009 at 9:17 AM, Claus Ibsen <cl...@gmail.com> wrote:
>> Hi
>>
>> I have committed first cut of the new Async API to Camel trunk.
>> See ticket CAMEL-1572 for svn revision and details.
>>
>> The remaining work:
>> - Migrate MulticastProcessor
> DONE
>
>> - Remove last piece of old API classes (2 interfaces)
> DONE
>
>> - Introduce Async DSL to replace Thread DSL and with clear intention
>> of turning route into async mode and support thread pools using JDK
>> executor service.
>> - Support Jetty continuation with async
>>
>
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Status update
On Wed, May 6, 2009 at 9:17 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> I have committed first cut of the new Async API to Camel trunk.
> See ticket CAMEL-1572 for svn revision and details.
>
> The remaining work:
> - Migrate MulticastProcessor
DONE
> - Remove last piece of old API classes (2 interfaces)
DONE
> - Introduce Async DSL to replace Thread DSL and with clear intention
> of turning route into async mode and support thread pools using JDK
> executor service.
> - Support Jetty continuation with async
>
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
I have committed first cut of the new Async API to Camel trunk.
See ticket CAMEL-1572 for svn revision and details.
The remaining work:
- Migrate MulticastProcessor
- Remove last piece of old API classes (2 interfaces)
- Introduce Async DSL to replace Thread DSL and with clear intention
of turning route into async mode and support thread pools using JDK
executor service.
- Support Jetty continuation with async
Should we mark the old API in Camel 1.x as @deprecated?
On Tue, May 5, 2009 at 4:48 PM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> Just wanted to keep you posted.
>
> Had some more work done to replace the old async API with the new API
> in camel-core.
> I basically got it all done except the MuticastProcessor as it would
> take a bit more time to get done correctly.
> I am thinking it should leverage the JDK CompletionService or the like
> for keeping track on completed tasks.
>
> But for now I wanted to get the other parts working, and it looks good.
>
> The Thread DSL is to be replaced with a new Async DSL, so that is why
> I am deleting it from the code base.
>
> And I am going to merge the async producer template into the regular
> producer template so we have a single combined template for all
> producer sync and async.
> Its about 10 more methods.
>
>
>
>
> On Tue, May 5, 2009 at 11:05 AM, Claus Ibsen <cl...@gmail.com> wrote:
>> Hi
>>
>> I had some more time to work on this.
>> I have attached a 2nd patch on the JIRA ticket 1572.
>>
>> This time the patch is only focusing on the AsyncProducerTemplate. I
>> have added more methods to cover the most common methods use in
>> producer template.
>> With this an end user can initiate and async routing in Camel.
>>
>> If the user wants to use the regular Camel API (lookup endpoint,
>> create exchange, create producer, process etc.) there is a helper
>> method on AsyncProcessorHelper to initiate an async route as well.
>>
>>
>> Any thoughts?
>>
>>
>>
>> On Fri, May 1, 2009 at 11:53 AM, Claus Ibsen <cl...@gmail.com> wrote:
>>> I have committed a patch to trunk:
>>> that resolves #1 (partly), #2 and #4
>>>
>>> In #1 there are still two classes that needs a cleanup as well.
>>> However they are core pieces in Camel, the Pipeline and
>>> MulticastProcessor so they are left as is. When we have a new Async
>>> API in place these classes can be fixed as well.
>>>
>>> I have also attached a prototype for a patch for the new async API
>>> experiment I have done. See CAMEL-1572.
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>> Apache Camel Reference Card:
>>> http://refcardz.dzone.com/refcardz/enterprise-integration
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>> Apache Camel Reference Card:
>> http://refcardz.dzone.com/refcardz/enterprise-integration
>> Interview with me:
>> http://architects.dzone.com/articles/interview-claus-ibsen-about?mz=7893-progress
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
> Apache Camel Reference Card:
> http://refcardz.dzone.com/refcardz/enterprise-integration
> Interview with me:
> http://architects.dzone.com/articles/interview-claus-ibsen-about?mz=7893-progress
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Interview with me:
http://architects.dzone.com/articles/interview-claus-ibsen-about?mz=7893-progress
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Just wanted to keep you posted.
Had some more work done to replace the old async API with the new API
in camel-core.
I basically got it all done except the MuticastProcessor as it would
take a bit more time to get done correctly.
I am thinking it should leverage the JDK CompletionService or the like
for keeping track on completed tasks.
But for now I wanted to get the other parts working, and it looks good.
The Thread DSL is to be replaced with a new Async DSL, so that is why
I am deleting it from the code base.
And I am going to merge the async producer template into the regular
producer template so we have a single combined template for all
producer sync and async.
Its about 10 more methods.
On Tue, May 5, 2009 at 11:05 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> I had some more time to work on this.
> I have attached a 2nd patch on the JIRA ticket 1572.
>
> This time the patch is only focusing on the AsyncProducerTemplate. I
> have added more methods to cover the most common methods use in
> producer template.
> With this an end user can initiate and async routing in Camel.
>
> If the user wants to use the regular Camel API (lookup endpoint,
> create exchange, create producer, process etc.) there is a helper
> method on AsyncProcessorHelper to initiate an async route as well.
>
>
> Any thoughts?
>
>
>
> On Fri, May 1, 2009 at 11:53 AM, Claus Ibsen <cl...@gmail.com> wrote:
>> I have committed a patch to trunk:
>> that resolves #1 (partly), #2 and #4
>>
>> In #1 there are still two classes that needs a cleanup as well.
>> However they are core pieces in Camel, the Pipeline and
>> MulticastProcessor so they are left as is. When we have a new Async
>> API in place these classes can be fixed as well.
>>
>> I have also attached a prototype for a patch for the new async API
>> experiment I have done. See CAMEL-1572.
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>> Apache Camel Reference Card:
>> http://refcardz.dzone.com/refcardz/enterprise-integration
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
> Apache Camel Reference Card:
> http://refcardz.dzone.com/refcardz/enterprise-integration
> Interview with me:
> http://architects.dzone.com/articles/interview-claus-ibsen-about?mz=7893-progress
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Interview with me:
http://architects.dzone.com/articles/interview-claus-ibsen-about?mz=7893-progress
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
I had some more time to work on this.
I have attached a 2nd patch on the JIRA ticket 1572.
This time the patch is only focusing on the AsyncProducerTemplate. I
have added more methods to cover the most common methods use in
producer template.
With this an end user can initiate and async routing in Camel.
If the user wants to use the regular Camel API (lookup endpoint,
create exchange, create producer, process etc.) there is a helper
method on AsyncProcessorHelper to initiate an async route as well.
Any thoughts?
On Fri, May 1, 2009 at 11:53 AM, Claus Ibsen <cl...@gmail.com> wrote:
> I have committed a patch to trunk:
> that resolves #1 (partly), #2 and #4
>
> In #1 there are still two classes that needs a cleanup as well.
> However they are core pieces in Camel, the Pipeline and
> MulticastProcessor so they are left as is. When we have a new Async
> API in place these classes can be fixed as well.
>
> I have also attached a prototype for a patch for the new async API
> experiment I have done. See CAMEL-1572.
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
> Apache Camel Reference Card:
> http://refcardz.dzone.com/refcardz/enterprise-integration
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Interview with me:
http://architects.dzone.com/articles/interview-claus-ibsen-about?mz=7893-progress
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
I have committed a patch to trunk:
that resolves #1 (partly), #2 and #4
In #1 there are still two classes that needs a cleanup as well.
However they are core pieces in Camel, the Pipeline and
MulticastProcessor so they are left as is. When we have a new Async
API in place these classes can be fixed as well.
I have also attached a prototype for a patch for the new async API
experiment I have done. See CAMEL-1572.
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
I have uploaded a 2nd patch at CAMEL-1572 - (the b patch)
I got the async API prototype also.
On Thu, Apr 30, 2009 at 5:39 PM, Claus Ibsen <cl...@gmail.com> wrote:
> On Thu, Apr 30, 2009 at 4:00 PM, Willem Jiang <wi...@gmail.com> wrote:
>> How about introducing Async requestBody API to the template?
>> It just return a Future object, then we can turn the value of Future
>> into what we want.
> Good idea. In fact I have done that already.
>
> Future<Object> requestAsyncBody(String endpoint, Object body);
>
> But I wanted to start with a "raw" Exchange at first.
> The idea is to have similar methods as the regular producer template,
> so you can shoot in headers as well.
>
> There is though something to think a bit more about is the exchange
> patterns for inOnly how that can still have the need to return the
> Future object as you might want to know when or if the task is done,
> even thought there is no response for you.
>
> So instead of void as return its probably gonna be something like Future.
>
> I will later add another patch at the CAMEL-1572 ticket so you can
> take a closer look at my experiments.
>
>
>
>
>
>>
>> Willem
>>
>> Claus Ibsen wrote:
>>> Hi
>>>
>>> Its me again. Yeah I am due for a run in due time, but just wanted to
>>> demo something that is either powerful or scary
>>>
>>> The code below uses the same route. But as we request a body and we
>>> have declared we want the response as a String.class (the last
>>> parameter)
>>> Then Camel is "clever" or "scary" to use the Future to wait and get
>>> the result and convert the body to the desired type, eg a String.
>>>
>>> So the routing is really divided into sync/async but the end user sees
>>> it as a single sync.
>>>
>>> public void testAsyncRouteWithTypeConverted() throws Exception {
>>> MockEndpoint mock = getMockEndpoint("mock:result");
>>> mock.expectedBodiesReceived("Bye World");
>>>
>>> // send a request reply to the direct start endpoint, but will use
>>> // future type converter that will wait for the response
>>> String response = template.requestBody("direct:start",
>>> "Hello", String.class);
>>> assertEquals("Bye World", response);
>>>
>>> assertMockEndpointsSatisfied();
>>> }
>>>
>>> So whats next is that you can just pass in the Future<String.class> to
>>> instruct Camel that you want the future handle back
>>> and that it should return a String as the response.
>>>
>>> Then Camel should be able to take it from there. With the future
>>> handle the caller gets back in control when the routes hits the
>>> async() DSL.
>>> And can do other work as he like.
>>>
>>> And when he want the response, he just:
>>> Future<String> future = template.requestBody("direct:start", "Hello",
>>> Future<String>.class);
>>>
>>> String response = future.get();
>>>
>>> But then I guess this is impossible due to type erasure in java generics :(
>>> No its not!!!
>>>
>>> Okay time to hit the streets
>>>
>>>
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
> Apache Camel Reference Card:
> http://refcardz.dzone.com/refcardz/enterprise-integration
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Apr 30, 2009 at 4:00 PM, Willem Jiang <wi...@gmail.com> wrote:
> How about introducing Async requestBody API to the template?
> It just return a Future object, then we can turn the value of Future
> into what we want.
Good idea. In fact I have done that already.
Future<Object> requestAsyncBody(String endpoint, Object body);
But I wanted to start with a "raw" Exchange at first.
The idea is to have similar methods as the regular producer template,
so you can shoot in headers as well.
There is though something to think a bit more about is the exchange
patterns for inOnly how that can still have the need to return the
Future object as you might want to know when or if the task is done,
even thought there is no response for you.
So instead of void as return its probably gonna be something like Future.
I will later add another patch at the CAMEL-1572 ticket so you can
take a closer look at my experiments.
>
> Willem
>
> Claus Ibsen wrote:
>> Hi
>>
>> Its me again. Yeah I am due for a run in due time, but just wanted to
>> demo something that is either powerful or scary
>>
>> The code below uses the same route. But as we request a body and we
>> have declared we want the response as a String.class (the last
>> parameter)
>> Then Camel is "clever" or "scary" to use the Future to wait and get
>> the result and convert the body to the desired type, eg a String.
>>
>> So the routing is really divided into sync/async but the end user sees
>> it as a single sync.
>>
>> public void testAsyncRouteWithTypeConverted() throws Exception {
>> MockEndpoint mock = getMockEndpoint("mock:result");
>> mock.expectedBodiesReceived("Bye World");
>>
>> // send a request reply to the direct start endpoint, but will use
>> // future type converter that will wait for the response
>> String response = template.requestBody("direct:start",
>> "Hello", String.class);
>> assertEquals("Bye World", response);
>>
>> assertMockEndpointsSatisfied();
>> }
>>
>> So whats next is that you can just pass in the Future<String.class> to
>> instruct Camel that you want the future handle back
>> and that it should return a String as the response.
>>
>> Then Camel should be able to take it from there. With the future
>> handle the caller gets back in control when the routes hits the
>> async() DSL.
>> And can do other work as he like.
>>
>> And when he want the response, he just:
>> Future<String> future = template.requestBody("direct:start", "Hello",
>> Future<String>.class);
>>
>> String response = future.get();
>>
>> But then I guess this is impossible due to type erasure in java generics :(
>> No its not!!!
>>
>> Okay time to hit the streets
>>
>>
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Willem Jiang <wi...@gmail.com>.
How about introducing Async requestBody API to the template?
It just return a Future object, then we can turn the value of Future
into what we want.
Willem
Claus Ibsen wrote:
> Hi
>
> Its me again. Yeah I am due for a run in due time, but just wanted to
> demo something that is either powerful or scary
>
> The code below uses the same route. But as we request a body and we
> have declared we want the response as a String.class (the last
> parameter)
> Then Camel is "clever" or "scary" to use the Future to wait and get
> the result and convert the body to the desired type, eg a String.
>
> So the routing is really divided into sync/async but the end user sees
> it as a single sync.
>
> public void testAsyncRouteWithTypeConverted() throws Exception {
> MockEndpoint mock = getMockEndpoint("mock:result");
> mock.expectedBodiesReceived("Bye World");
>
> // send a request reply to the direct start endpoint, but will use
> // future type converter that will wait for the response
> String response = template.requestBody("direct:start",
> "Hello", String.class);
> assertEquals("Bye World", response);
>
> assertMockEndpointsSatisfied();
> }
>
> So whats next is that you can just pass in the Future<String.class> to
> instruct Camel that you want the future handle back
> and that it should return a String as the response.
>
> Then Camel should be able to take it from there. With the future
> handle the caller gets back in control when the routes hits the
> async() DSL.
> And can do other work as he like.
>
> And when he want the response, he just:
> Future<String> future = template.requestBody("direct:start", "Hello",
> Future<String>.class);
>
> String response = future.get();
>
> But then I guess this is impossible due to type erasure in java generics :(
> No its not!!!
>
> Okay time to hit the streets
>
>
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Its me again. Yeah I am due for a run in due time, but just wanted to
demo something that is either powerful or scary
The code below uses the same route. But as we request a body and we
have declared we want the response as a String.class (the last
parameter)
Then Camel is "clever" or "scary" to use the Future to wait and get
the result and convert the body to the desired type, eg a String.
So the routing is really divided into sync/async but the end user sees
it as a single sync.
public void testAsyncRouteWithTypeConverted() throws Exception {
MockEndpoint mock = getMockEndpoint("mock:result");
mock.expectedBodiesReceived("Bye World");
// send a request reply to the direct start endpoint, but will use
// future type converter that will wait for the response
String response = template.requestBody("direct:start",
"Hello", String.class);
assertEquals("Bye World", response);
assertMockEndpointsSatisfied();
}
So whats next is that you can just pass in the Future<String.class> to
instruct Camel that you want the future handle back
and that it should return a String as the response.
Then Camel should be able to take it from there. With the future
handle the caller gets back in control when the routes hits the
async() DSL.
And can do other work as he like.
And when he want the response, he just:
Future<String> future = template.requestBody("direct:start", "Hello",
Future<String>.class);
String response = future.get();
But then I guess this is impossible due to type erasure in java generics :(
No its not!!!
Okay time to hit the streets
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Apr 30, 2009 at 4:05 PM, Willem Jiang <wi...@gmail.com> wrote:
> Hi Claus,
>
> Does the async() DSL will fork a thread for the further processing?
> What's the difference between the async() DSL and send the message to a
> seda endpoint?
Yes it forks a thread. Its the JDK Executor that handles this. That is
the beauty as we leverage the concurrency API in JDK.
The work I have done today is just experimental and thus nothing is
settled in stone. I just wanted to share what I have "played with"
today.
The different is that with async() DSL we leverage 100% the JDK
concurrency API for async callbacks. So async() DSL will return a
handle to the callback known as the Future handle from the JDK itself.
Then the original caller can use this handle to get the result.
With seda we do not leverage the JDK async API in the same sense. We
send a copy of the exchange to a JDK queue that a consumer in another
thread is listening.
Another difference is the seda is an endpoint. And thus we can use it
with the dynamic recipient list EIP patterns and all the other
patterns that work with endpoints. The async() is a DSL that is fixed
in that sence it breaks the current processing of the Exchange and
returns a Future handle to the original caller. And spaws a thread and
continues routing the exchange. All using the JDK async API.
The code for the async() DSL is basically:
public void process(final Exchange exchange) throws Exception {
// let it execute async and return the Future
Callable<Exchange> task = new Callable<Exchange>() {
public Exchange call() throws Exception {
for (Processor output : outputs) {
output.process(exchange);
}
return exchange;
}
};
Future<Exchange> future = executor.submit(task);
// set the future as response so we can get this handle and
get the response later
exchange.getOut().setBody(future);
}
It was just an experiment as I wanted to see if it was possible to
change the current route into async without seda.
>
> Thanks,
>
> Willem
>
> Claus Ibsen wrote:
>> Hi
>>
>> Just another update on #3. I had some more fun with Camel and
>> introduced a async() DSL in the route, to turn the route into async
>> from the point forward.
>> The unit test code explains it, and gives a hint how we can leverage this.
>>
>> Any thoughts?
>>
>>
>>
>> public class AsyncRouteTest extends ContextTestSupport {
>>
>> public void testAsyncRoute() throws Exception {
>> MockEndpoint mock = getMockEndpoint("mock:result");
>> mock.expectedBodiesReceived("Bye World");
>>
>> // send a request reply to the direct start endpoint
>> Object out = template.requestBody("direct:start", "Hello");
>> // as it turns into a async route later we get a Future as response
>> assertIsInstanceOf(Future.class, out);
>>
>> // cast to future
>> Future future = (Future) out;
>>
>> System.out.println("Look ma I can do other stuff while the async runs");
>>
>> // and use future to get the response
>> Exchange response = (Exchange) future.get();
>>
>> // get the response from the OUT message
>> // TODO: add type converters so we can leverage them with
>> Future to get the
>> // body response more easily
>> assertEquals("Bye World", response.getOut().getBody());
>>
>> assertMockEndpointsSatisfied();
>> }
>>
>> @Override
>> protected RouteBuilder createRouteBuilder() throws Exception {
>> return new RouteBuilder() {
>> @Override
>> public void configure() throws Exception {
>> // we start this route async
>> from("direct:start")
>> // we play a bit with the message
>> .transform(body().append(" World"))
>> // now turn the route into async from this point forward
>> // the caller will have a Future<Exchange>
>> returned as response in OUT
>> // to be used to grap the async response when he
>> fell like it
>> .async()
>> // from this point forward this is the async route
>> doing its work
>> // so we do a bit of delay to simulate heavy work
>> that takes time
>> .delay(1000)
>> // and we also play with the message so we can
>> prepare a response
>> .process(new Processor() {
>> public void process(Exchange exchange) throws
>> Exception {
>> assertEquals("Hello World",
>> exchange.getIn().getBody());
>> exchange.getOut().setBody("Bye World");
>> }
>> // and we use mocks for unit testing
>> }).to("mock:result");
>> }
>> };
>> }
>>
>>
>>
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Willem Jiang <wi...@gmail.com>.
Hi Claus,
Does the async() DSL will fork a thread for the further processing?
What's the difference between the async() DSL and send the message to a
seda endpoint?
Thanks,
Willem
Claus Ibsen wrote:
> Hi
>
> Just another update on #3. I had some more fun with Camel and
> introduced a async() DSL in the route, to turn the route into async
> from the point forward.
> The unit test code explains it, and gives a hint how we can leverage this.
>
> Any thoughts?
>
>
>
> public class AsyncRouteTest extends ContextTestSupport {
>
> public void testAsyncRoute() throws Exception {
> MockEndpoint mock = getMockEndpoint("mock:result");
> mock.expectedBodiesReceived("Bye World");
>
> // send a request reply to the direct start endpoint
> Object out = template.requestBody("direct:start", "Hello");
> // as it turns into a async route later we get a Future as response
> assertIsInstanceOf(Future.class, out);
>
> // cast to future
> Future future = (Future) out;
>
> System.out.println("Look ma I can do other stuff while the async runs");
>
> // and use future to get the response
> Exchange response = (Exchange) future.get();
>
> // get the response from the OUT message
> // TODO: add type converters so we can leverage them with
> Future to get the
> // body response more easily
> assertEquals("Bye World", response.getOut().getBody());
>
> assertMockEndpointsSatisfied();
> }
>
> @Override
> protected RouteBuilder createRouteBuilder() throws Exception {
> return new RouteBuilder() {
> @Override
> public void configure() throws Exception {
> // we start this route async
> from("direct:start")
> // we play a bit with the message
> .transform(body().append(" World"))
> // now turn the route into async from this point forward
> // the caller will have a Future<Exchange>
> returned as response in OUT
> // to be used to grap the async response when he
> fell like it
> .async()
> // from this point forward this is the async route
> doing its work
> // so we do a bit of delay to simulate heavy work
> that takes time
> .delay(1000)
> // and we also play with the message so we can
> prepare a response
> .process(new Processor() {
> public void process(Exchange exchange) throws
> Exception {
> assertEquals("Hello World",
> exchange.getIn().getBody());
> exchange.getOut().setBody("Bye World");
> }
> // and we use mocks for unit testing
> }).to("mock:result");
> }
> };
> }
>
>
>
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Just another update on #3. I had some more fun with Camel and
introduced a async() DSL in the route, to turn the route into async
from the point forward.
The unit test code explains it, and gives a hint how we can leverage this.
Any thoughts?
public class AsyncRouteTest extends ContextTestSupport {
public void testAsyncRoute() throws Exception {
MockEndpoint mock = getMockEndpoint("mock:result");
mock.expectedBodiesReceived("Bye World");
// send a request reply to the direct start endpoint
Object out = template.requestBody("direct:start", "Hello");
// as it turns into a async route later we get a Future as response
assertIsInstanceOf(Future.class, out);
// cast to future
Future future = (Future) out;
System.out.println("Look ma I can do other stuff while the async runs");
// and use future to get the response
Exchange response = (Exchange) future.get();
// get the response from the OUT message
// TODO: add type converters so we can leverage them with
Future to get the
// body response more easily
assertEquals("Bye World", response.getOut().getBody());
assertMockEndpointsSatisfied();
}
@Override
protected RouteBuilder createRouteBuilder() throws Exception {
return new RouteBuilder() {
@Override
public void configure() throws Exception {
// we start this route async
from("direct:start")
// we play a bit with the message
.transform(body().append(" World"))
// now turn the route into async from this point forward
// the caller will have a Future<Exchange>
returned as response in OUT
// to be used to grap the async response when he
fell like it
.async()
// from this point forward this is the async route
doing its work
// so we do a bit of delay to simulate heavy work
that takes time
.delay(1000)
// and we also play with the message so we can
prepare a response
.process(new Processor() {
public void process(Exchange exchange) throws
Exception {
assertEquals("Hello World",
exchange.getIn().getBody());
exchange.getOut().setBody("Bye World");
}
// and we use mocks for unit testing
}).to("mock:result");
}
};
}
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
This is just a short update on:
Ad 3)
The new Async API
I started iTunes with some good tunes from my youth gone wild period,
so I had a crack at prototype of the async API.
Here is the first unit test demonstrating how it could work with the
JDK Future API (for real async)
public class DefaultAsyncProcessorTemplateTest extends ContextTestSupport {
public void testRequestAsync() throws Exception {
AsyncProducerTemplate async = new DefaultAsyncProducerTemplate(context);
async.start();
Exchange exchange = new DefaultExchange(context);
exchange.getIn().setBody("Hello");
Future<Exchange> future = async.requestAsync("direct:start", exchange);
long start = System.currentTimeMillis();
System.out.println("Look ma I can do other stuff while the async runs");
Exchange result = future.get();
long delta = System.currentTimeMillis() - start;
System.out.println("Damn that async takes time so I had to
wait " + delta + " ms.");
assertEquals("Hello World", result.getOut().getBody());
async.stop();
}
@Override
protected RouteBuilder createRouteBuilder() throws Exception {
return new RouteBuilder() {
@Override
public void configure() throws Exception {
from("direct:start")
.delay(2000)
.transform(body().append(" World")).to("mock:result");
}
};
}
}
And the console ouput when I run it
2009-04-30 10:43:56,585 [main ] INFO DefaultCamelContext
- Apache Camel (CamelContext:camel-1) started
2009-04-30 10:43:56,586 [main ] DEBUG
aultAsyncProcessorTemplateTest - Routing Rules are: ....
Look ma I can do other stuff while the async runs
2009-04-30 10:43:58,833 [pool-1-thread-1] DEBUG MockEndpoint
- mock:result >>>> 1 : Exchange[Message: Hello World] with
body: Hello World
Damn that async takes time so I had to wait 2134 ms.
2009-04-30 10:43:58,835 [main ] DEBUG
aultAsyncProcessorTemplateTest - tearDown test: testRequestAsync
On Thu, Apr 30, 2009 at 7:32 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> You might remember my previous topic about the AsyncProcessor.
> http://www.nabble.com/-DISCUSS---Camel-2.0---Internal-API-reworkings----Channel-and--AsyncProcessor-td23210093.html
>
>
> I had some more time to thinker about it and I believe I have a
> roadmap for a solution that will work nicely for all.
>
> 1) Internal house cleaning in camel-core
> 2) Fix as much of the current broken AsyncCallback
> 3) Introduce new async API in Camel 2.0
> 4) Mark the current AsyncCallback as @deprecated in Camel 2.0m2
>
>
> Ad 1)
> The AsyncProcessing internally in Camel is overused and it just makes
> it much more complex. By doing a house cleaning the codebase is much
> easier to work with and it works as before.
> In fact it allows us to do fix the broken AsyncCallback also so we can
> get notification when the async exchange is done.
>
>
> Ad 2)
> The current notification in Camel is in fact broken. You never get the
> expected notification when the async exchange is done. The idea of
> this API was to get 2 notifications
> - the 1st is when the original synch thread is done (the boolean is
> true), eg when you submit the job.
> - and then the 2nd when the async thread is done (the boolean is
> false) but this is broken in Camel and you will not receive it.
>
> And of top of this there are 2 flaws in the current API
> - the current code requires that you manaully add seda steps in the
> route to turn the route from sync into async. There was no code that
> did this "under the covers for you". However its doable and the in
> fact its possible to route async. But an improve API would be better.
> - The result of the async operation is not possible to get easily. No
> you dont have access to the exchange that was done async. And no
> exchange.getOut() will not give the result. What was missing in the
> AsyncCallback API was to pass in the Exchange as well.
>
>
> Ad 3)
> I would like to introduce a new Async API that is more similar to the
> JDK Future API. So you can use the Future get() to get hold of the
> result.
> This still needs some thinking and experimenting to see how its
> doable. But an API that generally is like the Future would be much
> better.
>
> Ad 4)
> I would also suggest that we (in Camel 2.0m2) mark AsyncCallback as
> @deprecated with a note that we have planned a new Async API for Camel
> 2.0
>
>
> Conclusion
> ========
> I would like to go forward and get #1, (#2 and #4 would be great too)
> committed into the codebase so we have a codebase in Camel that is
> much easier to maintain and for new users to understand when they look
> into the internals in Camel. And especially debugging and stepping
> through the processing of Exchanges is much easier. As its just
> regular method calls. And we get rid of the confusing code with what
> to do in the AsyncCallbacks that was invading the internals in Camel.
>
>
> Then I would start experimenting with the new API and report findings.
> Feedback here is much welcome.
>
>
> Any thoughts?
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
> Apache Camel Reference Card:
> http://refcardz.dzone.com/refcardz/enterprise-integration
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Re: Camel 2.0 Async Findings - Roadmap to a solution
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
I have created a ticket to track this
https://issues.apache.org/activemq/browse/CAMEL-1572
And I have attached a patch to address item #1, #2 and #4.
All unit tests passes (core, components, tests, etc.)
On Thu, Apr 30, 2009 at 7:32 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> You might remember my previous topic about the AsyncProcessor.
> http://www.nabble.com/-DISCUSS---Camel-2.0---Internal-API-reworkings----Channel-and--AsyncProcessor-td23210093.html
>
>
> I had some more time to thinker about it and I believe I have a
> roadmap for a solution that will work nicely for all.
>
> 1) Internal house cleaning in camel-core
> 2) Fix as much of the current broken AsyncCallback
> 3) Introduce new async API in Camel 2.0
> 4) Mark the current AsyncCallback as @deprecated in Camel 2.0m2
>
>
> Ad 1)
> The AsyncProcessing internally in Camel is overused and it just makes
> it much more complex. By doing a house cleaning the codebase is much
> easier to work with and it works as before.
> In fact it allows us to do fix the broken AsyncCallback also so we can
> get notification when the async exchange is done.
>
>
> Ad 2)
> The current notification in Camel is in fact broken. You never get the
> expected notification when the async exchange is done. The idea of
> this API was to get 2 notifications
> - the 1st is when the original synch thread is done (the boolean is
> true), eg when you submit the job.
> - and then the 2nd when the async thread is done (the boolean is
> false) but this is broken in Camel and you will not receive it.
>
> And of top of this there are 2 flaws in the current API
> - the current code requires that you manaully add seda steps in the
> route to turn the route from sync into async. There was no code that
> did this "under the covers for you". However its doable and the in
> fact its possible to route async. But an improve API would be better.
> - The result of the async operation is not possible to get easily. No
> you dont have access to the exchange that was done async. And no
> exchange.getOut() will not give the result. What was missing in the
> AsyncCallback API was to pass in the Exchange as well.
>
>
> Ad 3)
> I would like to introduce a new Async API that is more similar to the
> JDK Future API. So you can use the Future get() to get hold of the
> result.
> This still needs some thinking and experimenting to see how its
> doable. But an API that generally is like the Future would be much
> better.
>
> Ad 4)
> I would also suggest that we (in Camel 2.0m2) mark AsyncCallback as
> @deprecated with a note that we have planned a new Async API for Camel
> 2.0
>
>
> Conclusion
> ========
> I would like to go forward and get #1, (#2 and #4 would be great too)
> committed into the codebase so we have a codebase in Camel that is
> much easier to maintain and for new users to understand when they look
> into the internals in Camel. And especially debugging and stepping
> through the processing of Exchanges is much easier. As its just
> regular method calls. And we get rid of the confusing code with what
> to do in the AsyncCallbacks that was invading the internals in Camel.
>
>
> Then I would start experimenting with the new API and report findings.
> Feedback here is much welcome.
>
>
> Any thoughts?
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
> Apache Camel Reference Card:
> http://refcardz.dzone.com/refcardz/enterprise-integration
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration