You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Raul Kripalani <ra...@evosent.com> on 2013/02/19 01:21:03 UTC
[DISCUSS] Camel 3.0 - Core of the routing engine
Hello team,
We use a recursive model in our routing engine to chain processors with one
another to process exchanges.
This results in lengthy stacktraces and increased memory usage due to local
variable retention for longer than strictly needed, IMHO. This recent
question on Stack Overflow is a typical (short!) stacktrace:
http://stackoverflow.com/questions/14734983/camel-ftp-intermittent-issue.
Debugging
it can be daunting for users.
Moreover many infrastructural Processors are woven implicitly along the
processor chain. Maybe this logic should belong elsewhere (the routing core
itself?).
Now that the Camel routing model is consolidated, can we start thinking
about moving towards an iterative routing approach? I feel the recursive
approach worked wonders when Camel was still a baby and the architecture
was heavily evolving: basically any processor, at any point, could do
anything to the exchange. And everything was a processor. Flexible and
versatile!
But now that the concepts are well rooted, I feel we need to formally
define "the routing process", rather than leaving it all up to processors
to be assembled in the right order.
I realize my proposal may sound somewhat abstract at this point, but before
going to further length, I want to gauge if my concern is shared.
What do you think?
Raúl.
Re: [DISCUSS] Camel 3.0 - Core of the routing engine
Posted by Łukasz Dywicki <lu...@code-house.org>.
Raul you have huge
+1
from me on this idea.
Basically I was thinking about idea of "logical" modules for Camel 3.0 which could modify routing chain, property configurations. Overall purpose is to get rid from core all magic which was built around PropertiesComponent/TraceInterceptor and so on (don't know why it still remains in core). I can't easily count how many instanceof statements we have in code.
The "module" thing came to me after using Jackson 2.0. An example module which enables Hibernate integration can be found on github [1].
[1] https://github.com/FasterXML/jackson-module-hibernate
Kind regards,
Łukasz Dywicki
--
luke@code-house.org
Twitter: ldywicki
Blog: http://dywicki.pl
Code-House - http://code-house.org
Wiadomość napisana przez Christian Schneider <ch...@die-schneider.net> w dniu 19 lut 2013, o godz. 08:20:
> +1
>
> I think one of the main things camel is missing is a routing engine.
> Like you wrote, currently each processor calls the next. Aspects like
> asynchronous behaviour, transaction and security are also
> implemented as processors.
>
> I think what we need is a runtime model of a camel route that represents
> the route without aspects. The aspects should then be executed
> by the routing engine without blowing up this model.
>
> This change is pretty severe though. I am not sure if we can refactor
> the camel core and keep it even somewhat compatible.
> I also fear that once we start these changes there is no way back and we
> might risk having an instable branch for a long time. So we have
> to reduce this risk in some way.
>
> Christian
>
> On 19.02.2013 01:21, Raul Kripalani wrote:
>> Hello team,
>>
>> We use a recursive model in our routing engine to chain processors with one
>> another to process exchanges.
>>
>> This results in lengthy stacktraces and increased memory usage due to local
>> variable retention for longer than strictly needed, IMHO. This recent
>> question on Stack Overflow is a typical (short!) stacktrace:
>> http://stackoverflow.com/questions/14734983/camel-ftp-intermittent-issue.
>> Debugging
>> it can be daunting for users.
>>
>> Moreover many infrastructural Processors are woven implicitly along the
>> processor chain. Maybe this logic should belong elsewhere (the routing core
>> itself?).
>>
>> Now that the Camel routing model is consolidated, can we start thinking
>> about moving towards an iterative routing approach? I feel the recursive
>> approach worked wonders when Camel was still a baby and the architecture
>> was heavily evolving: basically any processor, at any point, could do
>> anything to the exchange. And everything was a processor. Flexible and
>> versatile!
>>
>> But now that the concepts are well rooted, I feel we need to formally
>> define "the routing process", rather than leaving it all up to processors
>> to be assembled in the right order.
>>
>> I realize my proposal may sound somewhat abstract at this point, but before
>> going to further length, I want to gauge if my concern is shared.
>>
>> What do you think?
>>
>> Raúl.
>>
>
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
Re: [DISCUSS] Camel 3.0 - Core of the routing engine
Posted by Raul Kripalani <ra...@evosent.com>.
I'll create a section in the evening with references to the associated
ideas.
How are we handling the prototyping of ideas? Collaborate on code, reviews,
(virtual) pair programming if we have a joint championship, etc. SVN is
inflexible, but GitHub has the social element we need.
Working on branches is the way to go, but "one idea = one branch" seems
naïve because ideas can easily be interrelated. Shall we form "clusters" of
related ideas and spawn off a branch for each cluster?
For example, this core re-engineering + the two ideas that Claus mentioned
can form an "idea cluster" (IC). The champions of the cluster can then
create a branch each if they wish, merging onto the IC branch, and
ultimately onto trunk if the implementation is approved by the community.
Regards,
*Raúl Kripalani*
Apache Camel Committer
Enterprise Architect, Program Manager, Open Source Integration specialist
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk <http://twitter.com/raulvk>
On Tue, Feb 19, 2013 at 9:31 AM, Claus Ibsen <cl...@gmail.com> wrote:
> On Tue, Feb 19, 2013 at 8:20 AM, Christian Schneider
> <ch...@die-schneider.net> wrote:
> > +1
> >
> > I think one of the main things camel is missing is a routing engine.
> > Like you wrote, currently each processor calls the next. Aspects like
> > asynchronous behaviour, transaction and security are also
> > implemented as processors.
> >
> > I think what we need is a runtime model of a camel route that represents
> > the route without aspects. The aspects should then be executed
> > by the routing engine without blowing up this model.
> >
> > This change is pretty severe though. I am not sure if we can refactor
> > the camel core and keep it even somewhat compatible.
> > I also fear that once we start these changes there is no way back and we
> > might risk having an instable branch for a long time. So we have
> > to reduce this risk in some way.
> >
>
>
> Its been on the Camel 3.0 roadmap for a long time
>
> Though its heading caption may have been chosen a better wording than
> - More flexible routes at runtime
> http://camel.apache.org/camel-30-ideas.html
>
> We had it as target for a long time, but as you said it safter to work
> on this in a major release than on the 2.x architecture.
> And hence why its not been done yet.
>
> And in term of the model, then we have missing pieces about the
> inteceptors/onExceptions and whatnot. This is also on the roadmap for
> Camel 3.0
> - Add OnException, Interceptor, etc. to JAXB model for a
> CamelContextDefinition
>
>
> > Christian
> >
> > On 19.02.2013 01:21, Raul Kripalani wrote:
> >> Hello team,
> >>
> >> We use a recursive model in our routing engine to chain processors with
> one
> >> another to process exchanges.
> >>
> >> This results in lengthy stacktraces and increased memory usage due to
> local
> >> variable retention for longer than strictly needed, IMHO. This recent
> >> question on Stack Overflow is a typical (short!) stacktrace:
> >>
> http://stackoverflow.com/questions/14734983/camel-ftp-intermittent-issue.
> >> Debugging
> >> it can be daunting for users.
> >>
> >> Moreover many infrastructural Processors are woven implicitly along the
> >> processor chain. Maybe this logic should belong elsewhere (the routing
> core
> >> itself?).
> >>
> >> Now that the Camel routing model is consolidated, can we start thinking
> >> about moving towards an iterative routing approach? I feel the recursive
> >> approach worked wonders when Camel was still a baby and the architecture
> >> was heavily evolving: basically any processor, at any point, could do
> >> anything to the exchange. And everything was a processor. Flexible and
> >> versatile!
> >>
> >> But now that the concepts are well rooted, I feel we need to formally
> >> define "the routing process", rather than leaving it all up to
> processors
> >> to be assembled in the right order.
> >>
> >> I realize my proposal may sound somewhat abstract at this point, but
> before
> >> going to further length, I want to gauge if my concern is shared.
> >>
> >> What do you think?
> >>
> >> Raúl.
> >>
> >
> >
> > --
> > Christian Schneider
> > http://www.liquid-reality.de
> >
> > Open Source Architect
> > http://www.talend.com
> >
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> FuseSource is now part of Red Hat
> Email: cibsen@redhat.com
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
Re: [DISCUSS] Camel 3.0 - Core of the routing engine
Posted by Raul Kripalani <ra...@evosent.com>.
Hi,
It took me a while to find the time, but this idea is now logged in
the Camel 3.0 ideas page. We'll spin off a dedicated page as the
definition continues evolving.
BTW - I also added a ToC to the page to simplify browsing.
Regards,
Raúl Kripalani | Apache Camel Committer | Enterprise Architect,
Program Manager, Open Source Integration specialist
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk
On Tue, Feb 19, 2013 at 9:49 PM, Christian Schneider
<ch...@die-schneider.net> wrote:
> I also thought about a future routing engine some time ago and I think your
> ideas make sense.
> When we remove the possibility of routing steps to simply call processors
> then we have to reflect the possible
> branches in the model itself. So it is pretty natural to introduce something
> like a RoutingDecider.
>
> We have to experiment a bit to see how to implement the different model
> elements camel currently provides
> but generally what you wrote looks good to me.
>
> For the crosscutting concerns I think we could define something like aspects
> that the routing engine then applies at runtime.
> This will make it much simpler and straightforward to implement them
>
> Christian
>
>
> Am 19.02.2013 22:08, schrieb Raul Kripalani:
>>
>> It's a big undertaking but I feel we're at the right moment to redesign
>> the
>> core of Camel, make it leaner and pull in the routing concepts into the
>> API
>> model itself. It's basically now or never.
>>
>> One of the cornerstones of what I have in mind is to diversify the concept
>> of the Processor into its several specialisations, e.g. by introducing the
>> concept of a RoutingDecider, Actions, etc.
>>
>> For instance, EIPs like ChoiceProcessor and FilterProcessors could
>> implement RoutingDecider. Instead of actually invoking the next processor
>> (which is what they do now), they would return it to the Routing Core
>> (RC).
>> The RC would be in charge of actually invoking the processor.
>>
>> LogProcessor, ConvertBodyTo, Delayer, etc. would be Actions: they execute
>> one scoped operation and they return the Exchange to the RC.
>>
>> Cross-cutting concerns like Instrumenting (InstrumentationProcessor), Unit
>> of Work Management, DelegateAsyncProcessor, etc. would belong somehow in
>> the RC itself.
>>
>> Error handlers would just be a referred to, rather than woven at several
>> steps. If a step in the routing chain throws an Exception, the RC would
>> invoke the Error Handling "box".
>>
>> To me, each route would be an instantiation of a RoutingCore, where the
>> direct: endpoint is capable of bridging RoutingCores together.
>>
>> I hope this makes sense to you as much as it does to me ;)
>>
>> Regards,
>>
>> *Raúl Kripalani*
>>
>> Apache Camel Committer
>> Enterprise Architect, Program Manager, Open Source Integration specialist
>> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
>> http://blog.raulkr.net | twitter: @raulvk <http://twitter.com/raulvk>
>>
>>
>> On Tue, Feb 19, 2013 at 7:34 PM, Hadrian Zbarcea <hz...@gmail.com>
>> wrote:
>>
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> Talend Application Integration Division http://www.talend.com
>
Re: [DISCUSS] Camel 3.0 - Core of the routing engine
Posted by Christian Schneider <ch...@die-schneider.net>.
I also thought about a future routing engine some time ago and I think
your ideas make sense.
When we remove the possibility of routing steps to simply call
processors then we have to reflect the possible
branches in the model itself. So it is pretty natural to introduce
something like a RoutingDecider.
We have to experiment a bit to see how to implement the different model
elements camel currently provides
but generally what you wrote looks good to me.
For the crosscutting concerns I think we could define something like
aspects that the routing engine then applies at runtime.
This will make it much simpler and straightforward to implement them
Christian
Am 19.02.2013 22:08, schrieb Raul Kripalani:
> It's a big undertaking but I feel we're at the right moment to redesign the
> core of Camel, make it leaner and pull in the routing concepts into the API
> model itself. It's basically now or never.
>
> One of the cornerstones of what I have in mind is to diversify the concept
> of the Processor into its several specialisations, e.g. by introducing the
> concept of a RoutingDecider, Actions, etc.
>
> For instance, EIPs like ChoiceProcessor and FilterProcessors could
> implement RoutingDecider. Instead of actually invoking the next processor
> (which is what they do now), they would return it to the Routing Core (RC).
> The RC would be in charge of actually invoking the processor.
>
> LogProcessor, ConvertBodyTo, Delayer, etc. would be Actions: they execute
> one scoped operation and they return the Exchange to the RC.
>
> Cross-cutting concerns like Instrumenting (InstrumentationProcessor), Unit
> of Work Management, DelegateAsyncProcessor, etc. would belong somehow in
> the RC itself.
>
> Error handlers would just be a referred to, rather than woven at several
> steps. If a step in the routing chain throws an Exception, the RC would
> invoke the Error Handling "box".
>
> To me, each route would be an instantiation of a RoutingCore, where the
> direct: endpoint is capable of bridging RoutingCores together.
>
> I hope this makes sense to you as much as it does to me ;)
>
> Regards,
>
> *Raúl Kripalani*
> Apache Camel Committer
> Enterprise Architect, Program Manager, Open Source Integration specialist
> http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
> http://blog.raulkr.net | twitter: @raulvk <http://twitter.com/raulvk>
>
> On Tue, Feb 19, 2013 at 7:34 PM, Hadrian Zbarcea <hz...@gmail.com> wrote:
>
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
Talend Application Integration Division http://www.talend.com
Re: [DISCUSS] Camel 3.0 - Core of the routing engine
Posted by Raul Kripalani <ra...@evosent.com>.
It's a big undertaking but I feel we're at the right moment to redesign the
core of Camel, make it leaner and pull in the routing concepts into the API
model itself. It's basically now or never.
One of the cornerstones of what I have in mind is to diversify the concept
of the Processor into its several specialisations, e.g. by introducing the
concept of a RoutingDecider, Actions, etc.
For instance, EIPs like ChoiceProcessor and FilterProcessors could
implement RoutingDecider. Instead of actually invoking the next processor
(which is what they do now), they would return it to the Routing Core (RC).
The RC would be in charge of actually invoking the processor.
LogProcessor, ConvertBodyTo, Delayer, etc. would be Actions: they execute
one scoped operation and they return the Exchange to the RC.
Cross-cutting concerns like Instrumenting (InstrumentationProcessor), Unit
of Work Management, DelegateAsyncProcessor, etc. would belong somehow in
the RC itself.
Error handlers would just be a referred to, rather than woven at several
steps. If a step in the routing chain throws an Exception, the RC would
invoke the Error Handling "box".
To me, each route would be an instantiation of a RoutingCore, where the
direct: endpoint is capable of bridging RoutingCores together.
I hope this makes sense to you as much as it does to me ;)
Regards,
*Raúl Kripalani*
Apache Camel Committer
Enterprise Architect, Program Manager, Open Source Integration specialist
http://about.me/raulkripalani | http://www.linkedin.com/in/raulkripalani
http://blog.raulkr.net | twitter: @raulvk <http://twitter.com/raulvk>
On Tue, Feb 19, 2013 at 7:34 PM, Hadrian Zbarcea <hz...@gmail.com> wrote:
> Comments inline.
> Cheers,
> Hadrian
>
>
> On 02/19/2013 01:09 PM, Claus Ibsen wrote:
>
>> On Tue, Feb 19, 2013 at 11:04 AM, Christian Schneider
>> <ch...@die-schneider.net> wrote:
>>
>>> Hi Claus,
>>>
>>> do you think it is possible to refactor camel core in this way (to
>>> implement a real routing engine)? After the refactorings I already did I
>>> am not sure it would work.
>>>
>>>
>> Yes it is possible. The main switch is to use a runtime iterator based
>> model, that decides at runtime
>> which next processor to invoke. Instead of the current "static" approach.
>>
>
> I know for a fact that it's much more complicated than that and I am more
> with Christian on that. While I think it's possible, it's far, far from
> just an iterator.
>
>
>
>>
>> Another aproach instead of refactoring would be to first define a new
>>> really slim API for components and implement it using the current
>>> camel-core.
>>> Then we would change each component to use the new API. When all
>>> components are switched it will be much easier to change the core as we
>>> do not have to
>>> refactor all components to make the build work again. We discussed this
>>> idea at last Apachecon with Guillaume, Christian Müller and Charles.
>>> Then it was mainly in the light of having a small self contained API but
>>> I think it would also help with the routing engine.
>>>
>>>
>> The change in the routing engine should be internal facing only, and
>> all the tests should ideally pass.
>> There may be a few gremlins in the tests that has some internal
>> assumptions. But any end user scenario
>> of unit tests should have the goal of pass as is. To ensure backwards
>> compatability.
>>
> Agree that the routing engine should be a change internal to the runtime.
> The tests we have are not unit tests and in principle I agree with the goal
> of them passing as is after the refactoring. However I believe that with
> the tests in the current shape it will be cost (time/energy) prohibitive to
> refactor the core.
>
>
>
>> So there is no need to change any unit tests. Also the unit tests we
>> have today serve as good use-cases
>> for people to understand how to use Camel and its routes / components.
>>
>
> Actually in my opinion changing the tests is an absolutee must for 3.0 to
> have a chance of being more than lipstick. If we give that up, we'll give
> up refactoring the api as well. That's what my crystal globe tells me.
>
>
>
>
>>
>>
>> Christian
>>>
>>> On 19.02.2013 10:31, Claus Ibsen wrote:
>>>
>>>>
>>>> Its been on the Camel 3.0 roadmap for a long time
>>>>
>>>> Though its heading caption may have been chosen a better wording than
>>>> - More flexible routes at runtime
>>>> http://camel.apache.org/camel-**30-ideas.html<http://camel.apache.org/camel-30-ideas.html>
>>>>
>>>> We had it as target for a long time, but as you said it safter to work
>>>> on this in a major release than on the 2.x architecture.
>>>> And hence why its not been done yet.
>>>>
>>>> And in term of the model, then we have missing pieces about the
>>>> inteceptors/onExceptions and whatnot. This is also on the roadmap for
>>>> Camel 3.0
>>>> - Add OnException, Interceptor, etc. to JAXB model for a
>>>> CamelContextDefinition
>>>>
>>>>
>>>>
>>> --
>>> Christian Schneider
>>> http://www.liquid-reality.de
>>>
>>> Open Source Architect
>>> http://www.talend.com
>>>
>>>
>>
>>
>>
Re: [DISCUSS] Camel 3.0 - Core of the routing engine
Posted by James Carman <jc...@carmanconsulting.com>.
I'm interested in this too! I'm here to help.
On Feb 19, 2013, at 3:44 PM, Christian Müller <ch...@gmail.com> wrote:
> Hadrian,
>
> did you found time to work on the new Camel components test kit?
> Should we plan a coding session in Portland?
>
> Best,
> Christian
>
> On Tue, Feb 19, 2013 at 8:34 PM, Hadrian Zbarcea <hz...@gmail.com> wrote:
>
>> Comments inline.
>> Cheers,
>> Hadrian
>>
>>
>> On 02/19/2013 01:09 PM, Claus Ibsen wrote:
>>
>>> On Tue, Feb 19, 2013 at 11:04 AM, Christian Schneider
>>> <ch...@die-schneider.net> wrote:
>>>
>>>> Hi Claus,
>>>>
>>>> do you think it is possible to refactor camel core in this way (to
>>>> implement a real routing engine)? After the refactorings I already did I
>>>> am not sure it would work.
>>>>
>>>>
>>> Yes it is possible. The main switch is to use a runtime iterator based
>>> model, that decides at runtime
>>> which next processor to invoke. Instead of the current "static" approach.
>>>
>>
>> I know for a fact that it's much more complicated than that and I am more
>> with Christian on that. While I think it's possible, it's far, far from
>> just an iterator.
>>
>>
>>
>>>
>>> Another aproach instead of refactoring would be to first define a new
>>>> really slim API for components and implement it using the current
>>>> camel-core.
>>>> Then we would change each component to use the new API. When all
>>>> components are switched it will be much easier to change the core as we
>>>> do not have to
>>>> refactor all components to make the build work again. We discussed this
>>>> idea at last Apachecon with Guillaume, Christian Müller and Charles.
>>>> Then it was mainly in the light of having a small self contained API but
>>>> I think it would also help with the routing engine.
>>>>
>>>>
>>> The change in the routing engine should be internal facing only, and
>>> all the tests should ideally pass.
>>> There may be a few gremlins in the tests that has some internal
>>> assumptions. But any end user scenario
>>> of unit tests should have the goal of pass as is. To ensure backwards
>>> compatability.
>>>
>> Agree that the routing engine should be a change internal to the runtime.
>> The tests we have are not unit tests and in principle I agree with the goal
>> of them passing as is after the refactoring. However I believe that with
>> the tests in the current shape it will be cost (time/energy) prohibitive to
>> refactor the core.
>>
>>
>>
>>> So there is no need to change any unit tests. Also the unit tests we
>>> have today serve as good use-cases
>>> for people to understand how to use Camel and its routes / components.
>>>
>>
>> Actually in my opinion changing the tests is an absolutee must for 3.0 to
>> have a chance of being more than lipstick. If we give that up, we'll give
>> up refactoring the api as well. That's what my crystal globe tells me.
>>
>>
>>
>>
>>>
>>>
>>> Christian
>>>>
>>>> On 19.02.2013 10:31, Claus Ibsen wrote:
>>>>
>>>>>
>>>>> Its been on the Camel 3.0 roadmap for a long time
>>>>>
>>>>> Though its heading caption may have been chosen a better wording than
>>>>> - More flexible routes at runtime
>>>>> http://camel.apache.org/camel-**30-ideas.html<http://camel.apache.org/camel-30-ideas.html>
>>>>>
>>>>> We had it as target for a long time, but as you said it safter to work
>>>>> on this in a major release than on the 2.x architecture.
>>>>> And hence why its not been done yet.
>>>>>
>>>>> And in term of the model, then we have missing pieces about the
>>>>> inteceptors/onExceptions and whatnot. This is also on the roadmap for
>>>>> Camel 3.0
>>>>> - Add OnException, Interceptor, etc. to JAXB model for a
>>>>> CamelContextDefinition
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Christian Schneider
>>>> http://www.liquid-reality.de
>>>>
>>>> Open Source Architect
>>>> http://www.talend.com
>>>>
>>>>
>>>
>>>
>>>
>
>
> --
Re: [DISCUSS] Camel 3.0 - Core of the routing engine
Posted by Christian Müller <ch...@gmail.com>.
Hadrian,
did you found time to work on the new Camel components test kit?
Should we plan a coding session in Portland?
Best,
Christian
On Tue, Feb 19, 2013 at 8:34 PM, Hadrian Zbarcea <hz...@gmail.com> wrote:
> Comments inline.
> Cheers,
> Hadrian
>
>
> On 02/19/2013 01:09 PM, Claus Ibsen wrote:
>
>> On Tue, Feb 19, 2013 at 11:04 AM, Christian Schneider
>> <ch...@die-schneider.net> wrote:
>>
>>> Hi Claus,
>>>
>>> do you think it is possible to refactor camel core in this way (to
>>> implement a real routing engine)? After the refactorings I already did I
>>> am not sure it would work.
>>>
>>>
>> Yes it is possible. The main switch is to use a runtime iterator based
>> model, that decides at runtime
>> which next processor to invoke. Instead of the current "static" approach.
>>
>
> I know for a fact that it's much more complicated than that and I am more
> with Christian on that. While I think it's possible, it's far, far from
> just an iterator.
>
>
>
>>
>> Another aproach instead of refactoring would be to first define a new
>>> really slim API for components and implement it using the current
>>> camel-core.
>>> Then we would change each component to use the new API. When all
>>> components are switched it will be much easier to change the core as we
>>> do not have to
>>> refactor all components to make the build work again. We discussed this
>>> idea at last Apachecon with Guillaume, Christian Müller and Charles.
>>> Then it was mainly in the light of having a small self contained API but
>>> I think it would also help with the routing engine.
>>>
>>>
>> The change in the routing engine should be internal facing only, and
>> all the tests should ideally pass.
>> There may be a few gremlins in the tests that has some internal
>> assumptions. But any end user scenario
>> of unit tests should have the goal of pass as is. To ensure backwards
>> compatability.
>>
> Agree that the routing engine should be a change internal to the runtime.
> The tests we have are not unit tests and in principle I agree with the goal
> of them passing as is after the refactoring. However I believe that with
> the tests in the current shape it will be cost (time/energy) prohibitive to
> refactor the core.
>
>
>
>> So there is no need to change any unit tests. Also the unit tests we
>> have today serve as good use-cases
>> for people to understand how to use Camel and its routes / components.
>>
>
> Actually in my opinion changing the tests is an absolutee must for 3.0 to
> have a chance of being more than lipstick. If we give that up, we'll give
> up refactoring the api as well. That's what my crystal globe tells me.
>
>
>
>
>>
>>
>> Christian
>>>
>>> On 19.02.2013 10:31, Claus Ibsen wrote:
>>>
>>>>
>>>> Its been on the Camel 3.0 roadmap for a long time
>>>>
>>>> Though its heading caption may have been chosen a better wording than
>>>> - More flexible routes at runtime
>>>> http://camel.apache.org/camel-**30-ideas.html<http://camel.apache.org/camel-30-ideas.html>
>>>>
>>>> We had it as target for a long time, but as you said it safter to work
>>>> on this in a major release than on the 2.x architecture.
>>>> And hence why its not been done yet.
>>>>
>>>> And in term of the model, then we have missing pieces about the
>>>> inteceptors/onExceptions and whatnot. This is also on the roadmap for
>>>> Camel 3.0
>>>> - Add OnException, Interceptor, etc. to JAXB model for a
>>>> CamelContextDefinition
>>>>
>>>>
>>>>
>>> --
>>> Christian Schneider
>>> http://www.liquid-reality.de
>>>
>>> Open Source Architect
>>> http://www.talend.com
>>>
>>>
>>
>>
>>
--
Re: [DISCUSS] Camel 3.0 - Core of the routing engine
Posted by Hadrian Zbarcea <hz...@gmail.com>.
Comments inline.
Cheers,
Hadrian
On 02/19/2013 01:09 PM, Claus Ibsen wrote:
> On Tue, Feb 19, 2013 at 11:04 AM, Christian Schneider
> <ch...@die-schneider.net> wrote:
>> Hi Claus,
>>
>> do you think it is possible to refactor camel core in this way (to
>> implement a real routing engine)? After the refactorings I already did I
>> am not sure it would work.
>>
>
> Yes it is possible. The main switch is to use a runtime iterator based
> model, that decides at runtime
> which next processor to invoke. Instead of the current "static" approach.
I know for a fact that it's much more complicated than that and I am
more with Christian on that. While I think it's possible, it's far, far
from just an iterator.
>
>
>> Another aproach instead of refactoring would be to first define a new
>> really slim API for components and implement it using the current
>> camel-core.
>> Then we would change each component to use the new API. When all
>> components are switched it will be much easier to change the core as we
>> do not have to
>> refactor all components to make the build work again. We discussed this
>> idea at last Apachecon with Guillaume, Christian Müller and Charles.
>> Then it was mainly in the light of having a small self contained API but
>> I think it would also help with the routing engine.
>>
>
> The change in the routing engine should be internal facing only, and
> all the tests should ideally pass.
> There may be a few gremlins in the tests that has some internal
> assumptions. But any end user scenario
> of unit tests should have the goal of pass as is. To ensure backwards
> compatability.
Agree that the routing engine should be a change internal to the
runtime. The tests we have are not unit tests and in principle I agree
with the goal of them passing as is after the refactoring. However I
believe that with the tests in the current shape it will be cost
(time/energy) prohibitive to refactor the core.
>
> So there is no need to change any unit tests. Also the unit tests we
> have today serve as good use-cases
> for people to understand how to use Camel and its routes / components.
Actually in my opinion changing the tests is an absolutee must for 3.0
to have a chance of being more than lipstick. If we give that up, we'll
give up refactoring the api as well. That's what my crystal globe tells me.
>
>
>
>> Christian
>>
>> On 19.02.2013 10:31, Claus Ibsen wrote:
>>>
>>> Its been on the Camel 3.0 roadmap for a long time
>>>
>>> Though its heading caption may have been chosen a better wording than
>>> - More flexible routes at runtime
>>> http://camel.apache.org/camel-30-ideas.html
>>>
>>> We had it as target for a long time, but as you said it safter to work
>>> on this in a major release than on the 2.x architecture.
>>> And hence why its not been done yet.
>>>
>>> And in term of the model, then we have missing pieces about the
>>> inteceptors/onExceptions and whatnot. This is also on the roadmap for
>>> Camel 3.0
>>> - Add OnException, Interceptor, etc. to JAXB model for a CamelContextDefinition
>>>
>>>
>>
>> --
>> Christian Schneider
>> http://www.liquid-reality.de
>>
>> Open Source Architect
>> http://www.talend.com
>>
>
>
>
Re: [DISCUSS] Camel 3.0 - Core of the routing engine
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Feb 19, 2013 at 11:04 AM, Christian Schneider
<ch...@die-schneider.net> wrote:
> Hi Claus,
>
> do you think it is possible to refactor camel core in this way (to
> implement a real routing engine)? After the refactorings I already did I
> am not sure it would work.
>
Yes it is possible. The main switch is to use a runtime iterator based
model, that decides at runtime
which next processor to invoke. Instead of the current "static" approach.
> Another aproach instead of refactoring would be to first define a new
> really slim API for components and implement it using the current
> camel-core.
> Then we would change each component to use the new API. When all
> components are switched it will be much easier to change the core as we
> do not have to
> refactor all components to make the build work again. We discussed this
> idea at last Apachecon with Guillaume, Christian Müller and Charles.
> Then it was mainly in the light of having a small self contained API but
> I think it would also help with the routing engine.
>
The change in the routing engine should be internal facing only, and
all the tests should ideally pass.
There may be a few gremlins in the tests that has some internal
assumptions. But any end user scenario
of unit tests should have the goal of pass as is. To ensure backwards
compatability.
So there is no need to change any unit tests. Also the unit tests we
have today serve as good use-cases
for people to understand how to use Camel and its routes / components.
> Christian
>
> On 19.02.2013 10:31, Claus Ibsen wrote:
>>
>> Its been on the Camel 3.0 roadmap for a long time
>>
>> Though its heading caption may have been chosen a better wording than
>> - More flexible routes at runtime
>> http://camel.apache.org/camel-30-ideas.html
>>
>> We had it as target for a long time, but as you said it safter to work
>> on this in a major release than on the 2.x architecture.
>> And hence why its not been done yet.
>>
>> And in term of the model, then we have missing pieces about the
>> inteceptors/onExceptions and whatnot. This is also on the roadmap for
>> Camel 3.0
>> - Add OnException, Interceptor, etc. to JAXB model for a CamelContextDefinition
>>
>>
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: [DISCUSS] Camel 3.0 - Core of the routing engine
Posted by Christian Schneider <ch...@die-schneider.net>.
Hi Claus,
do you think it is possible to refactor camel core in this way (to
implement a real routing engine)? After the refactorings I already did I
am not sure it would work.
Another aproach instead of refactoring would be to first define a new
really slim API for components and implement it using the current
camel-core.
Then we would change each component to use the new API. When all
components are switched it will be much easier to change the core as we
do not have to
refactor all components to make the build work again. We discussed this
idea at last Apachecon with Guillaume, Christian Müller and Charles.
Then it was mainly in the light of having a small self contained API but
I think it would also help with the routing engine.
Christian
On 19.02.2013 10:31, Claus Ibsen wrote:
>
> Its been on the Camel 3.0 roadmap for a long time
>
> Though its heading caption may have been chosen a better wording than
> - More flexible routes at runtime
> http://camel.apache.org/camel-30-ideas.html
>
> We had it as target for a long time, but as you said it safter to work
> on this in a major release than on the 2.x architecture.
> And hence why its not been done yet.
>
> And in term of the model, then we have missing pieces about the
> inteceptors/onExceptions and whatnot. This is also on the roadmap for
> Camel 3.0
> - Add OnException, Interceptor, etc. to JAXB model for a CamelContextDefinition
>
>
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: [DISCUSS] Camel 3.0 - Core of the routing engine
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Feb 19, 2013 at 8:20 AM, Christian Schneider
<ch...@die-schneider.net> wrote:
> +1
>
> I think one of the main things camel is missing is a routing engine.
> Like you wrote, currently each processor calls the next. Aspects like
> asynchronous behaviour, transaction and security are also
> implemented as processors.
>
> I think what we need is a runtime model of a camel route that represents
> the route without aspects. The aspects should then be executed
> by the routing engine without blowing up this model.
>
> This change is pretty severe though. I am not sure if we can refactor
> the camel core and keep it even somewhat compatible.
> I also fear that once we start these changes there is no way back and we
> might risk having an instable branch for a long time. So we have
> to reduce this risk in some way.
>
Its been on the Camel 3.0 roadmap for a long time
Though its heading caption may have been chosen a better wording than
- More flexible routes at runtime
http://camel.apache.org/camel-30-ideas.html
We had it as target for a long time, but as you said it safter to work
on this in a major release than on the 2.x architecture.
And hence why its not been done yet.
And in term of the model, then we have missing pieces about the
inteceptors/onExceptions and whatnot. This is also on the roadmap for
Camel 3.0
- Add OnException, Interceptor, etc. to JAXB model for a CamelContextDefinition
> Christian
>
> On 19.02.2013 01:21, Raul Kripalani wrote:
>> Hello team,
>>
>> We use a recursive model in our routing engine to chain processors with one
>> another to process exchanges.
>>
>> This results in lengthy stacktraces and increased memory usage due to local
>> variable retention for longer than strictly needed, IMHO. This recent
>> question on Stack Overflow is a typical (short!) stacktrace:
>> http://stackoverflow.com/questions/14734983/camel-ftp-intermittent-issue.
>> Debugging
>> it can be daunting for users.
>>
>> Moreover many infrastructural Processors are woven implicitly along the
>> processor chain. Maybe this logic should belong elsewhere (the routing core
>> itself?).
>>
>> Now that the Camel routing model is consolidated, can we start thinking
>> about moving towards an iterative routing approach? I feel the recursive
>> approach worked wonders when Camel was still a baby and the architecture
>> was heavily evolving: basically any processor, at any point, could do
>> anything to the exchange. And everything was a processor. Flexible and
>> versatile!
>>
>> But now that the concepts are well rooted, I feel we need to formally
>> define "the routing process", rather than leaving it all up to processors
>> to be assembled in the right order.
>>
>> I realize my proposal may sound somewhat abstract at this point, but before
>> going to further length, I want to gauge if my concern is shared.
>>
>> What do you think?
>>
>> Raúl.
>>
>
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen@redhat.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: [DISCUSS] Camel 3.0 - Core of the routing engine
Posted by Christian Schneider <ch...@die-schneider.net>.
+1
I think one of the main things camel is missing is a routing engine.
Like you wrote, currently each processor calls the next. Aspects like
asynchronous behaviour, transaction and security are also
implemented as processors.
I think what we need is a runtime model of a camel route that represents
the route without aspects. The aspects should then be executed
by the routing engine without blowing up this model.
This change is pretty severe though. I am not sure if we can refactor
the camel core and keep it even somewhat compatible.
I also fear that once we start these changes there is no way back and we
might risk having an instable branch for a long time. So we have
to reduce this risk in some way.
Christian
On 19.02.2013 01:21, Raul Kripalani wrote:
> Hello team,
>
> We use a recursive model in our routing engine to chain processors with one
> another to process exchanges.
>
> This results in lengthy stacktraces and increased memory usage due to local
> variable retention for longer than strictly needed, IMHO. This recent
> question on Stack Overflow is a typical (short!) stacktrace:
> http://stackoverflow.com/questions/14734983/camel-ftp-intermittent-issue.
> Debugging
> it can be daunting for users.
>
> Moreover many infrastructural Processors are woven implicitly along the
> processor chain. Maybe this logic should belong elsewhere (the routing core
> itself?).
>
> Now that the Camel routing model is consolidated, can we start thinking
> about moving towards an iterative routing approach? I feel the recursive
> approach worked wonders when Camel was still a baby and the architecture
> was heavily evolving: basically any processor, at any point, could do
> anything to the exchange. And everything was a processor. Flexible and
> versatile!
>
> But now that the concepts are well rooted, I feel we need to formally
> define "the routing process", rather than leaving it all up to processors
> to be assembled in the right order.
>
> I realize my proposal may sound somewhat abstract at this point, but before
> going to further length, I want to gauge if my concern is shared.
>
> What do you think?
>
> Raúl.
>
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: [DISCUSS] Camel 3.0 - Core of the routing engine
Posted by Christian Müller <ch...@gmail.com>.
+1
Sent from a mobile device
Am 19.02.2013 01:21 schrieb "Raul Kripalani" <ra...@evosent.com>:
> Hello team,
>
> We use a recursive model in our routing engine to chain processors with one
> another to process exchanges.
>
> This results in lengthy stacktraces and increased memory usage due to local
> variable retention for longer than strictly needed, IMHO. This recent
> question on Stack Overflow is a typical (short!) stacktrace:
> http://stackoverflow.com/questions/14734983/camel-ftp-intermittent-issue.
> Debugging
> it can be daunting for users.
>
> Moreover many infrastructural Processors are woven implicitly along the
> processor chain. Maybe this logic should belong elsewhere (the routing core
> itself?).
>
> Now that the Camel routing model is consolidated, can we start thinking
> about moving towards an iterative routing approach? I feel the recursive
> approach worked wonders when Camel was still a baby and the architecture
> was heavily evolving: basically any processor, at any point, could do
> anything to the exchange. And everything was a processor. Flexible and
> versatile!
>
> But now that the concepts are well rooted, I feel we need to formally
> define "the routing process", rather than leaving it all up to processors
> to be assembled in the right order.
>
> I realize my proposal may sound somewhat abstract at this point, but before
> going to further length, I want to gauge if my concern is shared.
>
> What do you think?
>
> Raúl.
>