You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Grzegorz Borkowski <gr...@gmail.com> on 2012/02/10 12:34:48 UTC
Question on the proper usage of Camel
Hi All,
In our project we've been using Camel for a year, but with very mixed
feelings. I'd like to ask you if the way we use it is correct - maybe our
problems come from the wrong usage of the tool. Camel experts, please share
your opinions on this. Below you'll find description of the project.
Out project is an integration tool, integrating 3 other systems, call them
A, B and C. We receive the message from system A, query for some data from
system B and send the message to system C. Camel seemed to be ideal tool
for such scenario.
In practice, there are couple of problems though.
1. It turned out that we don't use any typical communication channels to
integrate with the systems. All those systems provided us with a library
(jar) which is simply a stub, which presents us with an API to call remote
systems synchronously. Even if those stubs use JMS or other channels
underneath, we don't interact with them directly. The only thing we do, is
just calling simple synchronous Java method on the stub. Something similar
to the old Java RMI. Thus, the whole power of Camel components is of no
value for us.
2. There is one place where we use Camel JMS component - we have a test
input, for receiving test data, which uses JMS communication. So currently
this is the only place in the code we use Camel components for real
integration with external systems. We're also now developing additional
interface, and this will be the second entry point based on JMS. So those
are only places when we use Camel capabilities.
3. Inside the application, we have a workflow build on the Camel routers,
predicates, and processors. All of them are glued by the "direct"
components. So when XML message comes, we pass it to a router, which passes
it to a proper processor, then to next router etc. There are around 20
processors and 10 routers in the flow. In practice however, we see only one
advantage of using Camel for this: ability to generate visual diagram of
the flow (using camel:dot feature). There are many downsides though. First,
there are many couplings between the processors, so we end up passing tens
of headers in the messages between processors. We already call it
"header-oriented programming" or simply "header hell". It's a nightmare.
Then, it's horror to debug the code when for example the router passed the
message to the wrong path. If it was a normal java code written by us, with
"if then else", we could just put a breakpoint there and debug it. But in
camel, if you configure routers using the built-in predicates, like:
from(...).choide().when(header(..).isEqualsTo(...)).to(...) - then you're
not able to debug it easily, because the if-then-else logic at runtime is
performed by camel engine, not our code.
4. Using the "direct" component connections everywhere seems to be
extremely inefficient. Say I have a flow: message goes through 2 routers
and two processors, linked by direct components. If you generate stack
trace at the end of this path, you will see literally hundreds of camel
calls. Where do they come from? It's especially frustrating when you try to
debug it - it's infeasible. I hate debugging Spring proxies because they
insert about 5-10 framework method calls between my classes. Camel does the
same, but it uses hundreds of framework calls instead. The stack trace is
littered with DelegateAsyncProcessor and AsyncProcessorHelper calls. I
wouldn't really be surprised if I saw StackOverflowError somewhere in the
flow.
5. Testing the Camel flow is not a nice experience as well. For every test,
you have to set up those dummy endpoints, and record expected number of
incoming messages. Then, when there is an exception thrown in the flow,
instead of just breaking the test where it was thrown, we see the cryptic
message at the end "expected receiving 1 message, received 0". Then you
have to analyze the log to find what was the real reason. Also, Camel
waiting for the messages, make the tests much slower, because time is lost
on this waiting.
Thoughts? Recommendations? What do we do wrong?
Re: Question on the proper usage of Camel
Posted by Hadrian Zbarcea <hz...@gmail.com>.
I'll jump in and try to clarify a few things. From the subject of this
tread, it looks like your not quite sure if you're using Camel in the
intended way. You already got a few good suggestions. In particular,
Ashwin's 1b is what I would have recommended myself too. Yet, Denis
Krizanovic's suggestion seems to resonate with you more. I am not sure
what Denis' experience with Camel is, he only posted a question once in
the past on this users@ list. That said, Camel, like most of the other
technologies you use, is just a tool and it's up to you to decide if
it's the right tool for the job or not. I also understand your
frustrations, some things seem overly complicated in Camel. I myself
share a part of that frustration and agree that some things could be
simpler. However, there is a reason why Camel works the way it does
(with an emphasis on "works"), but if it doesn't fit your needs, you may
want to look at other choices.
Camel prides itself in simplifying integration. That is for the larger
integration application, which, in my experience, is pretty much any
serious application nowadays. It does so in two main ways:
1. Hiding/abstracting the underlying communication protocols and allow
one to use a uniform api (hence the 1b. proposal that is usually my
first choice for scenarios like the one you describe)
2. Probably even more important, is the use of EIPs that allow
developers to convey the intent behind the integration scenario (plus
they are already implemented and come ootb with the Camel distro). What
I mean by that is that a "filter(expression)" is (at least imo) far more
suggestive to those who *read* the code than an "if (condition)"
statement, especially coupled with other things from "setBody" to
"choice" or "routingSlip". It tells readers 'what' you're trying to do,
instead of 'how'.
Then there are other goodies like TypeConverters, support for multiple
languages, etc.
Another set of integration requirements that Camel excels at fulfilling
are related to change. That is not just the initial implementation of an
integration application, but the way the application is operated and
maintained and grows in time to address new requirements. All this comes
down to costs and risks and Camel is very good at adapting to such needs
and make things *look* quite simple (although under the hood, as you
correctly mentioned, it's quite complicated). Easily adapting to change
is a priority for Camel. Matt touched on that too. (Looks like that may
not be a concern for you though).
So... I am not sure I can offer more concrete advice, but:
1. If the only useful thing out of Camel is using the camel:dot diagrams
then it may not be "proper usage of Camel"
2. If there are many couplings between processors, and got into "header
hell" you may have a bigger problem with the architecture/design of your
app. That thing won't scale no matter what you do and that may not even
be your biggest problem.
3. The "direct" component connections are actually synchronous in-memory
calls and as such, quite efficient. However if you are using many of
those, again, I would revisit the design.
4. Testing requires a bit of experience indeed, but it's actually not
that hard and once you get used do it everything starts to make more sense.
5. A bit of an aside, addressing Denis' analogy with the String
manipulation calls. If the String in question is a search query, than
having its processing by the search engine distributed in the Cloud
using a bunch of algorithms and indexes will be a very complex task,
although both the input and the output are fairly simple (and the tool
for that kind of jobs would be Apache Lucene, not Camel). My only point,
again, is that Camel may or may not be the tool you want and that is up
to you to decide. The choice has a subjective component too.
Good luck with your choice, thanks for trying Camel and if you have any
other questions in the future, feel free to ask the community on this list.
I hope this helps, at least a tiny bit,
Hadrian
On 02/12/2012 05:50 PM, Grzegorz Borkowski wrote:
> Ioannis, Denis,
>
> Thanks for you input.
> What you say is exactly the same what I think now, that we tried to
> "overuse" Camel. We actually used it as a workflow engine for the message
> processing, instead of integration engine. Instead of using Java
> if-then-else logic inside a processor, we used Camel routers, end that's
> why we ended with 20 processors, where in fact it should be definitely less
> than 5.
> For the questions about throughput and external system outages: the load is
> really small in this system, and it's very unlikely to grow to the point
> where it's of some concern for us. When the systems B or C are down, we
> return immediate error message to the system A, but we also try to
> reprocess the message after some time (we use timers which trigger
> reprocessing).
>
> W dniu 12 lutego 2012 22:34 użytkownik Denis Krizanovic<
> denis.krizanovic@gmail.com> napisał:
>
>> If all your doing is using an API, and you don't really care what is
>> happening underneath, then I can see no need for Camel.
>>
>> It would be the equivalent of trying to use Camel to do a series of String
>> manipulation calls. For all you know, the String class makes remote calls
>> over proprietary Oracle protocols to their Cloud platform to offer you the
>> result of substring. Of course it doesn't, but the point is, the API
>> ensures you don't care, just like in your case.
>>
>> Seems to me that you're just importing a lot of conceptual overhead, (ie
>> having to understand Camel) for no reason.
>>
>> regards,
>> dk-
>>
>> ps. What is questionable in this case, is why someone would bother writing
>> a wrapper library these days.
>>
>>
>> On 10 February 2012 22:34, Grzegorz Borkowski<gr...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> In our project we've been using Camel for a year, but with very mixed
>>> feelings. I'd like to ask you if the way we use it is correct - maybe our
>>> problems come from the wrong usage of the tool. Camel experts, please
>> share
>>> your opinions on this. Below you'll find description of the project.
>>>
>>>
>>> Out project is an integration tool, integrating 3 other systems, call
>> them
>>> A, B and C. We receive the message from system A, query for some data
>> from
>>> system B and send the message to system C. Camel seemed to be ideal tool
>>> for such scenario.
>>> In practice, there are couple of problems though.
>>>
>>> 1. It turned out that we don't use any typical communication channels to
>>> integrate with the systems. All those systems provided us with a library
>>> (jar) which is simply a stub, which presents us with an API to call
>> remote
>>> systems synchronously. Even if those stubs use JMS or other channels
>>> underneath, we don't interact with them directly. The only thing we do,
>> is
>>> just calling simple synchronous Java method on the stub. Something
>> similar
>>> to the old Java RMI. Thus, the whole power of Camel components is of no
>>> value for us.
>>>
>>> 2. There is one place where we use Camel JMS component - we have a test
>>> input, for receiving test data, which uses JMS communication. So
>> currently
>>> this is the only place in the code we use Camel components for real
>>> integration with external systems. We're also now developing additional
>>> interface, and this will be the second entry point based on JMS. So those
>>> are only places when we use Camel capabilities.
>>>
>>> 3. Inside the application, we have a workflow build on the Camel routers,
>>> predicates, and processors. All of them are glued by the "direct"
>>> components. So when XML message comes, we pass it to a router, which
>> passes
>>> it to a proper processor, then to next router etc. There are around 20
>>> processors and 10 routers in the flow. In practice however, we see only
>> one
>>> advantage of using Camel for this: ability to generate visual diagram of
>>> the flow (using camel:dot feature). There are many downsides though.
>> First,
>>> there are many couplings between the processors, so we end up passing
>> tens
>>> of headers in the messages between processors. We already call it
>>> "header-oriented programming" or simply "header hell". It's a nightmare.
>>> Then, it's horror to debug the code when for example the router passed
>> the
>>> message to the wrong path. If it was a normal java code written by us,
>> with
>>> "if then else", we could just put a breakpoint there and debug it. But in
>>> camel, if you configure routers using the built-in predicates, like:
>>> from(...).choide().when(header(..).isEqualsTo(...)).to(...) - then you're
>>> not able to debug it easily, because the if-then-else logic at runtime is
>>> performed by camel engine, not our code.
>>>
>>> 4. Using the "direct" component connections everywhere seems to be
>>> extremely inefficient. Say I have a flow: message goes through 2 routers
>>> and two processors, linked by direct components. If you generate stack
>>> trace at the end of this path, you will see literally hundreds of camel
>>> calls. Where do they come from? It's especially frustrating when you try
>> to
>>> debug it - it's infeasible. I hate debugging Spring proxies because they
>>> insert about 5-10 framework method calls between my classes. Camel does
>> the
>>> same, but it uses hundreds of framework calls instead. The stack trace is
>>> littered with DelegateAsyncProcessor and AsyncProcessorHelper calls. I
>>> wouldn't really be surprised if I saw StackOverflowError somewhere in the
>>> flow.
>>>
>>> 5. Testing the Camel flow is not a nice experience as well. For every
>> test,
>>> you have to set up those dummy endpoints, and record expected number of
>>> incoming messages. Then, when there is an exception thrown in the flow,
>>> instead of just breaking the test where it was thrown, we see the cryptic
>>> message at the end "expected receiving 1 message, received 0". Then you
>>> have to analyze the log to find what was the real reason. Also, Camel
>>> waiting for the messages, make the tests much slower, because time is
>> lost
>>> on this waiting.
>>>
>>>
>>> Thoughts? Recommendations? What do we do wrong?
>>>
>>
>
--
Hadrian Zbarcea
Principal Software Architect
Talend, Inc
http://coders.talend.com/
http://camelbot.blogspot.com/
Re: Question on the proper usage of Camel
Posted by Grzegorz Borkowski <gr...@gmail.com>.
Ioannis, Denis,
Thanks for you input.
What you say is exactly the same what I think now, that we tried to
"overuse" Camel. We actually used it as a workflow engine for the message
processing, instead of integration engine. Instead of using Java
if-then-else logic inside a processor, we used Camel routers, end that's
why we ended with 20 processors, where in fact it should be definitely less
than 5.
For the questions about throughput and external system outages: the load is
really small in this system, and it's very unlikely to grow to the point
where it's of some concern for us. When the systems B or C are down, we
return immediate error message to the system A, but we also try to
reprocess the message after some time (we use timers which trigger
reprocessing).
W dniu 12 lutego 2012 22:34 użytkownik Denis Krizanovic <
denis.krizanovic@gmail.com> napisał:
> If all your doing is using an API, and you don't really care what is
> happening underneath, then I can see no need for Camel.
>
> It would be the equivalent of trying to use Camel to do a series of String
> manipulation calls. For all you know, the String class makes remote calls
> over proprietary Oracle protocols to their Cloud platform to offer you the
> result of substring. Of course it doesn't, but the point is, the API
> ensures you don't care, just like in your case.
>
> Seems to me that you're just importing a lot of conceptual overhead, (ie
> having to understand Camel) for no reason.
>
> regards,
> dk-
>
> ps. What is questionable in this case, is why someone would bother writing
> a wrapper library these days.
>
>
> On 10 February 2012 22:34, Grzegorz Borkowski <gr...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > In our project we've been using Camel for a year, but with very mixed
> > feelings. I'd like to ask you if the way we use it is correct - maybe our
> > problems come from the wrong usage of the tool. Camel experts, please
> share
> > your opinions on this. Below you'll find description of the project.
> >
> >
> > Out project is an integration tool, integrating 3 other systems, call
> them
> > A, B and C. We receive the message from system A, query for some data
> from
> > system B and send the message to system C. Camel seemed to be ideal tool
> > for such scenario.
> > In practice, there are couple of problems though.
> >
> > 1. It turned out that we don't use any typical communication channels to
> > integrate with the systems. All those systems provided us with a library
> > (jar) which is simply a stub, which presents us with an API to call
> remote
> > systems synchronously. Even if those stubs use JMS or other channels
> > underneath, we don't interact with them directly. The only thing we do,
> is
> > just calling simple synchronous Java method on the stub. Something
> similar
> > to the old Java RMI. Thus, the whole power of Camel components is of no
> > value for us.
> >
> > 2. There is one place where we use Camel JMS component - we have a test
> > input, for receiving test data, which uses JMS communication. So
> currently
> > this is the only place in the code we use Camel components for real
> > integration with external systems. We're also now developing additional
> > interface, and this will be the second entry point based on JMS. So those
> > are only places when we use Camel capabilities.
> >
> > 3. Inside the application, we have a workflow build on the Camel routers,
> > predicates, and processors. All of them are glued by the "direct"
> > components. So when XML message comes, we pass it to a router, which
> passes
> > it to a proper processor, then to next router etc. There are around 20
> > processors and 10 routers in the flow. In practice however, we see only
> one
> > advantage of using Camel for this: ability to generate visual diagram of
> > the flow (using camel:dot feature). There are many downsides though.
> First,
> > there are many couplings between the processors, so we end up passing
> tens
> > of headers in the messages between processors. We already call it
> > "header-oriented programming" or simply "header hell". It's a nightmare.
> > Then, it's horror to debug the code when for example the router passed
> the
> > message to the wrong path. If it was a normal java code written by us,
> with
> > "if then else", we could just put a breakpoint there and debug it. But in
> > camel, if you configure routers using the built-in predicates, like:
> > from(...).choide().when(header(..).isEqualsTo(...)).to(...) - then you're
> > not able to debug it easily, because the if-then-else logic at runtime is
> > performed by camel engine, not our code.
> >
> > 4. Using the "direct" component connections everywhere seems to be
> > extremely inefficient. Say I have a flow: message goes through 2 routers
> > and two processors, linked by direct components. If you generate stack
> > trace at the end of this path, you will see literally hundreds of camel
> > calls. Where do they come from? It's especially frustrating when you try
> to
> > debug it - it's infeasible. I hate debugging Spring proxies because they
> > insert about 5-10 framework method calls between my classes. Camel does
> the
> > same, but it uses hundreds of framework calls instead. The stack trace is
> > littered with DelegateAsyncProcessor and AsyncProcessorHelper calls. I
> > wouldn't really be surprised if I saw StackOverflowError somewhere in the
> > flow.
> >
> > 5. Testing the Camel flow is not a nice experience as well. For every
> test,
> > you have to set up those dummy endpoints, and record expected number of
> > incoming messages. Then, when there is an exception thrown in the flow,
> > instead of just breaking the test where it was thrown, we see the cryptic
> > message at the end "expected receiving 1 message, received 0". Then you
> > have to analyze the log to find what was the real reason. Also, Camel
> > waiting for the messages, make the tests much slower, because time is
> lost
> > on this waiting.
> >
> >
> > Thoughts? Recommendations? What do we do wrong?
> >
>
Re: Question on the proper usage of Camel
Posted by Denis Krizanovic <de...@gmail.com>.
If all your doing is using an API, and you don't really care what is
happening underneath, then I can see no need for Camel.
It would be the equivalent of trying to use Camel to do a series of String
manipulation calls. For all you know, the String class makes remote calls
over proprietary Oracle protocols to their Cloud platform to offer you the
result of substring. Of course it doesn't, but the point is, the API
ensures you don't care, just like in your case.
Seems to me that you're just importing a lot of conceptual overhead, (ie
having to understand Camel) for no reason.
regards,
dk-
ps. What is questionable in this case, is why someone would bother writing
a wrapper library these days.
On 10 February 2012 22:34, Grzegorz Borkowski <gr...@gmail.com> wrote:
> Hi All,
>
> In our project we've been using Camel for a year, but with very mixed
> feelings. I'd like to ask you if the way we use it is correct - maybe our
> problems come from the wrong usage of the tool. Camel experts, please share
> your opinions on this. Below you'll find description of the project.
>
>
> Out project is an integration tool, integrating 3 other systems, call them
> A, B and C. We receive the message from system A, query for some data from
> system B and send the message to system C. Camel seemed to be ideal tool
> for such scenario.
> In practice, there are couple of problems though.
>
> 1. It turned out that we don't use any typical communication channels to
> integrate with the systems. All those systems provided us with a library
> (jar) which is simply a stub, which presents us with an API to call remote
> systems synchronously. Even if those stubs use JMS or other channels
> underneath, we don't interact with them directly. The only thing we do, is
> just calling simple synchronous Java method on the stub. Something similar
> to the old Java RMI. Thus, the whole power of Camel components is of no
> value for us.
>
> 2. There is one place where we use Camel JMS component - we have a test
> input, for receiving test data, which uses JMS communication. So currently
> this is the only place in the code we use Camel components for real
> integration with external systems. We're also now developing additional
> interface, and this will be the second entry point based on JMS. So those
> are only places when we use Camel capabilities.
>
> 3. Inside the application, we have a workflow build on the Camel routers,
> predicates, and processors. All of them are glued by the "direct"
> components. So when XML message comes, we pass it to a router, which passes
> it to a proper processor, then to next router etc. There are around 20
> processors and 10 routers in the flow. In practice however, we see only one
> advantage of using Camel for this: ability to generate visual diagram of
> the flow (using camel:dot feature). There are many downsides though. First,
> there are many couplings between the processors, so we end up passing tens
> of headers in the messages between processors. We already call it
> "header-oriented programming" or simply "header hell". It's a nightmare.
> Then, it's horror to debug the code when for example the router passed the
> message to the wrong path. If it was a normal java code written by us, with
> "if then else", we could just put a breakpoint there and debug it. But in
> camel, if you configure routers using the built-in predicates, like:
> from(...).choide().when(header(..).isEqualsTo(...)).to(...) - then you're
> not able to debug it easily, because the if-then-else logic at runtime is
> performed by camel engine, not our code.
>
> 4. Using the "direct" component connections everywhere seems to be
> extremely inefficient. Say I have a flow: message goes through 2 routers
> and two processors, linked by direct components. If you generate stack
> trace at the end of this path, you will see literally hundreds of camel
> calls. Where do they come from? It's especially frustrating when you try to
> debug it - it's infeasible. I hate debugging Spring proxies because they
> insert about 5-10 framework method calls between my classes. Camel does the
> same, but it uses hundreds of framework calls instead. The stack trace is
> littered with DelegateAsyncProcessor and AsyncProcessorHelper calls. I
> wouldn't really be surprised if I saw StackOverflowError somewhere in the
> flow.
>
> 5. Testing the Camel flow is not a nice experience as well. For every test,
> you have to set up those dummy endpoints, and record expected number of
> incoming messages. Then, when there is an exception thrown in the flow,
> instead of just breaking the test where it was thrown, we see the cryptic
> message at the end "expected receiving 1 message, received 0". Then you
> have to analyze the log to find what was the real reason. Also, Camel
> waiting for the messages, make the tests much slower, because time is lost
> on this waiting.
>
>
> Thoughts? Recommendations? What do we do wrong?
>
Re: Question on the proper usage of Camel
Posted by Nestor Urquiza <ne...@gmail.com>.
Very interesting thread. I will subscribe to it in case we get more feedback
but I can see extremely valuable points here. Camel as any other tool can be
miss used.
This thread has inspired me to
http://thinkinginsoftware.blogspot.com/2012/05/on-bpm-workflow-documentations-and-ba.html
blog once again about BPM, Workflow and Orchestration.
--
View this message in context: http://camel.465427.n5.nabble.com/Question-on-the-proper-usage-of-Camel-tp5472181p5709293.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Question on the proper usage of Camel
Posted by Ashwin Karpe <ak...@fusesource.com>.
Grzegorz,
A final words on stack traces and the how/why you see them....
This requires a deeper and nuanced understanding of how/why Camel and routes
work the way they do. The route is not a single threaded "if then else"
statement which can simply throw an exception when it occurs and bail...
Every segment of the route in DSL is decoupled and delegates (sync/async) to
the next segment along the route. The benefits of that are threads can be
most efficiently allocated to where they are most needed from a processing
standpoint at a given point in time. This is why Camel its extremely fast
and efficient.
An metaphor for this would be in case of a sudden data burst the camel route
would resemble " a Python gobbling a deer" (sorry about the visuals :)). As
the data burst moves through the route, camel in parallel adapts threads in
the most efficient way to different route segments and deals with the load
very well and is not easily overwhelmed.
However, the side effect of this is that when exceptions occurs, stack
unwinds look more voluminous and worrisome than they actually are. What is
more, there is no chance of a stack overflow since the exceptions originate
from different segments of the route and help from a traceability
standpoint. The only way a stack would overflow would be if the error were
to occur more than 32 levels deep in Java nested procedure calls. This is
simply not the case. If you look at your stack trace, you will see that each
of the exception is being thrown from different and each route segment where
the error was seen to originate from... More importantly these are not
nested procedure calls (the DelegateAsyncProcessor itself is a clue).
I hope this clarifies things. Having worked with Camel for a long time, call
me biased, but this is not an area I would go looking for humps and beat up
the poor beast of burden :)
Cheers,
Ashwin...
-----
---------------------------------------------------------
Ashwin Karpe
Apache Camel Committer & Sr Principal Consultant
FUSESource (a Progress Software Corporation subsidiary)
http://fusesource.com
Blog: http://opensourceknowledge.blogspot.com
---------------------------------------------------------
--
View this message in context: http://camel.465427.n5.nabble.com/Question-on-the-proper-usage-of-Camel-tp5472181p5489369.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Question on the proper usage of Camel
Posted by Grzegorz Borkowski <gr...@gmail.com>.
Thanks for the comments from all of you, now we have to analyze all of this
and possibly refactor our code, based on your feedback. Thanks for help,
Greg
W dniu 13 lutego 2012 10:13 użytkownik Ioannis Canellos
<io...@gmail.com>napisał:
> Grezgorz,
>
> Allow me to clarify that I did not mean that you are "overusing" camel. Or
> at least I don't consider building 20 processors Camel overuse. I think
> that you are just making your life harder, regardless of camel.
> For instance in a plain java application, building & wiring 20 classes to
> perform a single unit of work is a bit unnecessary.
>
> Now back to camel. The challenges in an integration project is not just
> being able interact with an external system. There are a lot more as I
> pointed in my original response. So, even if you don't need the components,
> camel has a lot more to offer that may not be visible at first.
>
> Ioannis Canellos
> FuseSource
>
> Blog: http://iocanel.blogspot.com
> Apache Karaf Committer & PMC
> Apache Camel Committer
> Apache ServiceMix Committer
> Apache Gora Committer
> Apache DirectMemory Committer
>
>
Re: Question on the proper usage of Camel
Posted by Ioannis Canellos <io...@gmail.com>.
Grezgorz,
Allow me to clarify that I did not mean that you are "overusing" camel. Or at least I don't consider building 20 processors Camel overuse. I think that you are just making your life harder, regardless of camel.
For instance in a plain java application, building & wiring 20 classes to perform a single unit of work is a bit unnecessary.
Now back to camel. The challenges in an integration project is not just being able interact with an external system. There are a lot more as I pointed in my original response. So, even if you don't need the components, camel has a lot more to offer that may not be visible at first.
Ioannis Canellos
FuseSource
Blog: http://iocanel.blogspot.com
Apache Karaf Committer & PMC
Apache Camel Committer
Apache ServiceMix Committer
Apache Gora Committer
Apache DirectMemory Committer
Re: Question on the proper usage of Camel
Posted by Christian Schneider <ch...@die-schneider.net>.
Am 11.02.2012 14:00, schrieb Grzegorz Borkowski:
> Ashwin, Matt,
>
> Thanks for you feedback. Here is some additional explanation of my points:
>
> 1. The external systems provide us with the ready-to-use
> proxies/stubs/facades. We just call the Java method on such stub and it
> does some remote calls internally, but it's beyond our control. What we
> see, is just a method to be called. Because of this, it turned out we don't
> need Camel components for communication with external systems. I don't see
> much advantage in wrapping those libraries in custom components...it's just
> more code to maintain, with no apparent benefit. So now we almost don't use
> Camel components. Then the question is, if you don't use Camel components
> (which are considered to be the biggest advantage of Camel), does it make
> sense to use Camel at all?
I think as long as you are using a library to access the external system
there is not much benefit in using Camel. One important question is though
if the concept if these libraries is a good solution. It means that the
developers of the external system have to give you a new version of the
library every time they update the system. This also means that you have
to update you own system at this time. I think this is a quite tight
coupling which
is not really desireable. The problem is multiplied when more than one
system uses the external system in this way. Another problem with these
libraries is that you
"import" all the technology the library uses into your own project. So
e.g. if they use spring then you will also have spring in your
depdencies. If you attach to two different
of those systems then this can be really difficult as they might require
to use different version of a lib at the same time.
So instead of your current aproach I would rather suggest to use kind of
a service oriented aproach. Instead of sharing a library you can define
the data format, transport and semantics to talk
to the external systems. This allows both sides to choose the technology
to provide or use the services freely. In this concept camel can help
you a lot for example you can use jaxb or json for the dataformat and
http or jms as transport. Camel frees you a lot from implementing these
classses.
Another thing you should consider is using pojo beans to implement your
business logic. Camel allows very good mapping to such beans and they
allow you to keep your business logic free of camel code. This makes
testing
the logic really easy as you can use pure junit tests instead of camel
tests for a lot of the cases.
>
> 2. In the current form, we use mainly Camel as a workflow engine, using
> only links between processor and routers. As Ashwin suggested, we could
> just directly couple the processors without without using "direct'
> components. Perhaps that's something to consider. But again the question
> is, is it a valid use case for Camel, if we just build the single-VM
> worfklow out of Camel processors and routers, with no components involved?
The camel routing is good if you deal with e.g. xml instead of java
objects. In your case where you already communicate with the other
system using objects I would rather code the routing in simple
java.
>
> 3. The direct endpoints efficiency - maybe the "efficiency" was not a best
> word, I didn't meant it's slow or so. It's just that if all you want is to
> pass a message from method X to method Y, you expect it to be easy
> operation. But then when you see the Camel stack trace, and you see several
> hundreds of the almost identical calls on the framework classes, then it's
> just suspicious, and you think something wrong is going on here. Why does
> Camel need hundred of calls on the stack to pass the message through a
> couple of processors/routers/direct components? From what I see, it's all
> littered with some async wrappers - i'm not getting the point of those
> async wrappers if all I need is a plain sync method call. Also, I haven't
> observed stack overflows yet - it's just that I'm very suspicious about the
> huge stacks produced by Camel.
Yes the async adapters in camel make it very difficult to debug and
probably also consume a lot of time for simple sync cases. I hope we can
address this issue in future camel versions. The reason why we have this
is that in many cases the async behaviour allows higher performance for
some cases. Of course async alone will already make debugging a lot
harder anyways.
> 4. Matt, I don't get your point about replacing direct calls with JMS. It
> sounds like a terrible over-engineering for me. I just need to call a Java
> method, why involve JMS?
I assume he suggests something similar like I described above. Trying te
decouple the systems. I think though that this does not work well with
the shared lib concept you currently have.
Christian
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
Talend Application Integration Division http://www.talend.com
Re: Question on the proper usage of Camel
Posted by Ioannis Canellos <io...@gmail.com>.
There are some points I miss here. From the initial description I understand that your use case is fairly simple: consume message from A, enrich / transform them using B & finally send them to C.
What I don't get is how you ended up having like 20 processors. The number sounds really big to me. I would understand having like 2 processors in a problem like the one describe, but 20 is really big and of course if you are trying to have them loosely coupled (which I don't see why) will eventually create a problem harder to solve than the initial one.
Since you have the integration libraries ready, beyond doubt is simpler to code just a class that calls the methods and not use any framework at all. It's simple, but also a bit "simplistic".
What I want to say is how are you going to handle in your code some typical integration problems?
i) You receive messages from A in higher rate than B responds, and you keep on piling messages in memory?
ii) You loose connection with service B, while receiving messages from A?
iii) You loose connection with service C?
iv) You go live and you find out that service B crashes when the load is increasing and you need some kind of throttling?
v) Specs change B responds asynchronously using a new service D (you need some correlation)?
Then you are going to need retry policies, maybe do some stuff async. On top of that you may also need some logic to filter out duplicate messages etc, throttling etc.
Are you going to code all these on your own?
So I would say that using 20 processors loosely coupled, might be an over engineering, but all the rest when we are talking about integration doesn't sound like over engineering to me.
Ioannis Canellos
FuseSource
Blog: http://iocanel.blogspot.com
Apache Karaf Committer & PMC
Apache Camel Committer
Apache ServiceMix Committer
Apache Gora Committer
Apache DirectMemory Committer
Re: Question on the proper usage of Camel
Posted by Matt Pavlovich <ma...@gmail.com>.
Grzegorz-
1. I was under the impression that you had a bunch of custom code doing
things like Web Service or database access. My contention was that
using Camel components to replace a lot of that boiler code is a big
win. There are still nice gains to be had from using Camel in your
scenario where you are talking with beans. The EIP patterns, the
management capabilities, etc.
2. Sure. There aren't any hard and fast rules about how you use it.
Using Camel is like making beer. Its only a bad idea if it doesn't
taste good ;-)
3. I don't think seeing a large stack trace is indicative of Camel
having suspect engineering. Its fast, it works and compared to most
other integration frameworks (especially the commercial ones!), its a
short stack trace :-).
4. I was just suggesting using queues as a way to shrink the custom code
stack I presumed from #1. I'm not suggesting over engineering, I don't
know enough about your specific implementation. For folks that are
moving from big stacks of EJBs that talk to 3rd party systems, using JMS
is one potential simplification approach. If your direct calls are
simple enough, no need to stick JMS in the middle.
Matt Pavlovich
On 2/11/12 7:00 AM, Grzegorz Borkowski wrote:
> Ashwin, Matt,
>
> Thanks for you feedback. Here is some additional explanation of my points:
>
> 1. The external systems provide us with the ready-to-use
> proxies/stubs/facades. We just call the Java method on such stub and it
> does some remote calls internally, but it's beyond our control. What we
> see, is just a method to be called. Because of this, it turned out we don't
> need Camel components for communication with external systems. I don't see
> much advantage in wrapping those libraries in custom components...it's just
> more code to maintain, with no apparent benefit. So now we almost don't use
> Camel components. Then the question is, if you don't use Camel components
> (which are considered to be the biggest advantage of Camel), does it make
> sense to use Camel at all?
>
> 2. In the current form, we use mainly Camel as a workflow engine, using
> only links between processor and routers. As Ashwin suggested, we could
> just directly couple the processors without without using "direct'
> components. Perhaps that's something to consider. But again the question
> is, is it a valid use case for Camel, if we just build the single-VM
> worfklow out of Camel processors and routers, with no components involved?
>
> 3. The direct endpoints efficiency - maybe the "efficiency" was not a best
> word, I didn't meant it's slow or so. It's just that if all you want is to
> pass a message from method X to method Y, you expect it to be easy
> operation. But then when you see the Camel stack trace, and you see several
> hundreds of the almost identical calls on the framework classes, then it's
> just suspicious, and you think something wrong is going on here. Why does
> Camel need hundred of calls on the stack to pass the message through a
> couple of processors/routers/direct components? From what I see, it's all
> littered with some async wrappers - i'm not getting the point of those
> async wrappers if all I need is a plain sync method call. Also, I haven't
> observed stack overflows yet - it's just that I'm very suspicious about the
> huge stacks produced by Camel.
>
> 4. Matt, I don't get your point about replacing direct calls with JMS. It
> sounds like a terrible over-engineering for me. I just need to call a Java
> method, why involve JMS?
>
Re: Question on the proper usage of Camel
Posted by Grzegorz Borkowski <gr...@gmail.com>.
Ashwin, Matt,
Thanks for you feedback. Here is some additional explanation of my points:
1. The external systems provide us with the ready-to-use
proxies/stubs/facades. We just call the Java method on such stub and it
does some remote calls internally, but it's beyond our control. What we
see, is just a method to be called. Because of this, it turned out we don't
need Camel components for communication with external systems. I don't see
much advantage in wrapping those libraries in custom components...it's just
more code to maintain, with no apparent benefit. So now we almost don't use
Camel components. Then the question is, if you don't use Camel components
(which are considered to be the biggest advantage of Camel), does it make
sense to use Camel at all?
2. In the current form, we use mainly Camel as a workflow engine, using
only links between processor and routers. As Ashwin suggested, we could
just directly couple the processors without without using "direct'
components. Perhaps that's something to consider. But again the question
is, is it a valid use case for Camel, if we just build the single-VM
worfklow out of Camel processors and routers, with no components involved?
3. The direct endpoints efficiency - maybe the "efficiency" was not a best
word, I didn't meant it's slow or so. It's just that if all you want is to
pass a message from method X to method Y, you expect it to be easy
operation. But then when you see the Camel stack trace, and you see several
hundreds of the almost identical calls on the framework classes, then it's
just suspicious, and you think something wrong is going on here. Why does
Camel need hundred of calls on the stack to pass the message through a
couple of processors/routers/direct components? From what I see, it's all
littered with some async wrappers - i'm not getting the point of those
async wrappers if all I need is a plain sync method call. Also, I haven't
observed stack overflows yet - it's just that I'm very suspicious about the
huge stacks produced by Camel.
4. Matt, I don't get your point about replacing direct calls with JMS. It
sounds like a terrible over-engineering for me. I just need to call a Java
method, why involve JMS?
Re: Question on the proper usage of Camel
Posted by Matt Pavlovich <ma...@gmail.com>.
Hi Grzegorz-
What you are experiencing is not uncommon for folks that are used to
100% Java-based coding. Its not really a technical issue, as much as it
is a conceptual one. I think one of the most important concepts to
think about is that the Camel approach encourages "configuring", vs
"coding". If you have a lot of existing Java code, it can be difficult
to make the decision to scrap it, but ultimately, I believe that you
will find you'll have far less project artifacts to maintain when using
Camel vs writing your own. Also, consider the upgrade cycle. When you
write your own transport connectors, you have to maintain and re-build
that code when upgrades for the underlying technology changes. With
Camel, you just have to keep up with any changes to the Camel API
(usually, very minor). As Camel upgrades, it brings the updated
dependencies with it-- at little or no impact to your Camel routes.
A lot of this comes down to maintainability and speed at which you can
implement an integration solution-- less project artifacts equals faster
delivery, easier maintenance.
1. If you are not using the components, you are missing out on one of
the most important benefit Camel brings. All that connection setup and
tear down code is redundant. If you are using JMS, HTTP, and databases,
your Java code is going to have a lot of redundancy to what Camel
provides for you. If you have some custom protocol that Camel doesn't
support yet, consider writing a small wrapper "entry service" that uses
a protocol that Camel does support. A Web Service, or a JMS queue. At
the end of the day, less code is always easier to support.
2. If you are making a lot of direct calls, you may consider the
benefits of having *more* JMS in the solution. JMS provides an easy and
reliable way to exchange information, and the guaranteed delivery, error
handling, HA and load balancing capabilites are provided for you, so you
don't have to worry about coding up process to account for those
scenarios. Using JMS would also provide you some statistics and metrics
"for free". Design your integrations around queues, and you can know
exactly how many requests were handled by process X.
3. Perhaps, replacing the "direct" calls with JMS Queues would help to
some degree. Also, look into the Camel tracer and log component. The
?showAll=true option on the log component is gold. It shows you all the
Camel exchange headers, properties and message payload. Great for
debugging.
4. I haven't done any specific load tests against the direct endpoint,
but I haven't seen anything that leads me to believe its horribly slow
or inefficient. A stack of method calls doesn't always necessarily mean
its inefficient.
5. See #3. One of the concepts I try to convey is that instead of
thinking of an integration from a big picture standpoint, think of it in
terms of discreet synchronous and asynchronous steps. These "steps" are
your routes. Leveraging queues and topics is a good way to provide
separation of steps and routes. Ultimately, there will be a few giant
routes that are just a pain. For those, leveraging the tracer, and log
steps makes debugging those a snap.
Hope this helps,
Matt Pavlovich
On 2/10/12 5:34 AM, Grzegorz Borkowski wrote:
> Hi All,
>
> In our project we've been using Camel for a year, but with very mixed
> feelings. I'd like to ask you if the way we use it is correct - maybe our
> problems come from the wrong usage of the tool. Camel experts, please share
> your opinions on this. Below you'll find description of the project.
>
>
> Out project is an integration tool, integrating 3 other systems, call them
> A, B and C. We receive the message from system A, query for some data from
> system B and send the message to system C. Camel seemed to be ideal tool
> for such scenario.
> In practice, there are couple of problems though.
>
> 1. It turned out that we don't use any typical communication channels to
> integrate with the systems. All those systems provided us with a library
> (jar) which is simply a stub, which presents us with an API to call remote
> systems synchronously. Even if those stubs use JMS or other channels
> underneath, we don't interact with them directly. The only thing we do, is
> just calling simple synchronous Java method on the stub. Something similar
> to the old Java RMI. Thus, the whole power of Camel components is of no
> value for us.
>
> 2. There is one place where we use Camel JMS component - we have a test
> input, for receiving test data, which uses JMS communication. So currently
> this is the only place in the code we use Camel components for real
> integration with external systems. We're also now developing additional
> interface, and this will be the second entry point based on JMS. So those
> are only places when we use Camel capabilities.
>
> 3. Inside the application, we have a workflow build on the Camel routers,
> predicates, and processors. All of them are glued by the "direct"
> components. So when XML message comes, we pass it to a router, which passes
> it to a proper processor, then to next router etc. There are around 20
> processors and 10 routers in the flow. In practice however, we see only one
> advantage of using Camel for this: ability to generate visual diagram of
> the flow (using camel:dot feature). There are many downsides though. First,
> there are many couplings between the processors, so we end up passing tens
> of headers in the messages between processors. We already call it
> "header-oriented programming" or simply "header hell". It's a nightmare.
> Then, it's horror to debug the code when for example the router passed the
> message to the wrong path. If it was a normal java code written by us, with
> "if then else", we could just put a breakpoint there and debug it. But in
> camel, if you configure routers using the built-in predicates, like:
> from(...).choide().when(header(..).isEqualsTo(...)).to(...) - then you're
> not able to debug it easily, because the if-then-else logic at runtime is
> performed by camel engine, not our code.
>
> 4. Using the "direct" component connections everywhere seems to be
> extremely inefficient. Say I have a flow: message goes through 2 routers
> and two processors, linked by direct components. If you generate stack
> trace at the end of this path, you will see literally hundreds of camel
> calls. Where do they come from? It's especially frustrating when you try to
> debug it - it's infeasible. I hate debugging Spring proxies because they
> insert about 5-10 framework method calls between my classes. Camel does the
> same, but it uses hundreds of framework calls instead. The stack trace is
> littered with DelegateAsyncProcessor and AsyncProcessorHelper calls. I
> wouldn't really be surprised if I saw StackOverflowError somewhere in the
> flow.
>
> 5. Testing the Camel flow is not a nice experience as well. For every test,
> you have to set up those dummy endpoints, and record expected number of
> incoming messages. Then, when there is an exception thrown in the flow,
> instead of just breaking the test where it was thrown, we see the cryptic
> message at the end "expected receiving 1 message, received 0". Then you
> have to analyze the log to find what was the real reason. Also, Camel
> waiting for the messages, make the tests much slower, because time is lost
> on this waiting.
>
>
> Thoughts? Recommendations? What do we do wrong?
>
Re: Question on the proper usage of Camel
Posted by Ashwin Karpe <ak...@fusesource.com>.
Hi,
Having read your message, there are several things you mention and in the
absence of specific architectural details, deployment and route details it
is hard to answer them unambiguously.
In any case, I will try to address some things you bring up with some of the
same generalities.
1. It turned out that we don't use any typical communication channels to
integrate with the systems. All those systems provided us with a library
(jar) which is simply a stub, which presents us with an API to call remote
systems synchronously. Even if those stubs use JMS or other channels
underneath, we don't interact with them directly. The only thing we do, is
just calling simple synchronous Java method on the stub. Something similar
to the old Java RMI. Thus, the whole power of Camel components is of no
value for us.
>>> If System A is a JMS Queue/Topic, unless the jar does something over and
>>> above a simple consume/subscribe, the Camel JMS component should be able
>>> to consume just as well. If however, the message sent by System A is
>>> sent to a non-JMS/Customized JMS Listener, there are 2 choices
a> Write a bean that sets up a listener and gets a message from System
A and pushes it downstream via a camel route (with a Direct, SEDA/VM
endpoint or any other endpoint).
b> Develop a custom Camel component that wraps your jar and acts as a
Consumer/Producer to Systems A, B and C
2. There is one place where we use Camel JMS component - we have a test
input, for receiving test data, which uses JMS communication. So currently
this is the only place in the code we use Camel components for real
integration with external systems. We're also now developing additional
interface, and this will be the second entry point based on JMS. So those
are only places when we use Camel capabilities.
>>>> At the end of the day, you know your system best. Whether to use or not
>>>> to use Camel is your call to make. This forum can help address your
>>>> technical difficulties but not give you a sales pitch on whether/why
>>>> you should/not use Camel ;). The more context and specific detail you
>>>> provide, the better the quality of answers will be...
3. Inside the application, we have a workflow build on the Camel routers,
predicates, and processors. All of them are glued by the "direct"
components. So when XML message comes, we pass it to a router, which passes
it to a proper processor, then to next router etc. There are around 20
processors and 10 routers in the flow. In practice however, we see only one
advantage of using Camel for this: ability to generate visual diagram of
the flow (using camel:dot feature). There are many downsides though. First,
there are many couplings between the processors, so we end up passing tens
of headers in the messages between processors. We already call it
"header-oriented programming" or simply "header hell". It's a nightmare.
Then, it's horror to debug the code when for example the router passed the
message to the wrong path. If it was a normal java code written by us, with
"if then else", we could just put a breakpoint there and debug it. But in
camel, if you configure routers using the built-in predicates, like:
from(...).choide().when(header(..).isEqualsTo(...)).to(...) - then you're
not able to debug it easily, because the if-then-else logic at runtime is
performed by camel engine, not our code.
>>> Please find my observations below...
a> You do not need to do header oriented programming in Camel at all.
You can get all the info you need from data itself if it is available and/or
infer/compute the value of headers if they are not available.
b> You also do not need to use camel DSL code to make your choices.
You can make these choices in Processors(Java code) as well. Please check
out the Camel Processors to do all your choice related work in Java... This
will simplify routes and allow you all the fine grained logging support you
need. It will also give you all the control you desire...
c> It is possible to write very elegant routes and processors and it
is equally possible to write atrocious and ugly routes with poorly crafted
processors and deployment methodologies. This is where you may need to
re-visit your application to see if this is indeed the case.
4. Using the "direct" component connections everywhere seems to be
extremely inefficient. Say I have a flow: message goes through 2 routers
and two processors, linked by direct components. If you generate stack
trace at the end of this path, you will see literally hundreds of camel
calls. Where do they come from? It's especially frustrating when you try to
debug it - it's infeasible. I hate debugging Spring proxies because they
insert about 5-10 framework method calls between my classes. Camel does the
same, but it uses hundreds of framework calls instead. The stack trace is
littered with DelegateAsyncProcessor and AsyncProcessorHelper calls. I
wouldn't really be surprised if I saw StackOverflowError somewhere in the
flow.
>>>> Now that is a very, let us say, "direct" statement ;). If you look at
>>>> the direct component source code, you will find it to be a simple
>>>> in-memory invocation (procedure call) without using queues or any
>>>> intermediary elements. This is why it works only in the same JVM. Now I
>>>> could write a route in several ways (2 given below for e.g.
1>from(a).to(b).to(c);
2>from(a).to(direct:x);
from(direct:x).to(b).to(direct:y);
from(direct:y).to(c);
Note that in the second example, the direct is a simple indirection to
another route segment and its only reason to be is that there may be many
other routes that push into direct:x or direct:y... Otherwise it is
processing overhead...
As for your question about stack traces and DelegateAsyncProcessors, this is
necessary for optimal thread performance and efficient processing along
every element of the route without locking up threads as a message flows
along the route. If you do see a StackOverflowError please do log an JIRA
entry for the same. If not, then you need check the lowest stack trace
message to get the exact nature of the error. This is a good thing from a
traceability standpoint. Since the throwable is being picked up and
re-thrown from several entities in the code, you know exactly how to
reproduce the problem and where to look for issues.
5. Testing the Camel flow is not a nice experience as well. For every test,
you have to set up those dummy endpoints, and record expected number of
incoming messages. Then, when there is an exception thrown in the flow,
instead of just breaking the test where it was thrown, we see the cryptic
message at the end "expected receiving 1 message, received 0". Then you
have to analyze the log to find what was the real reason. Also, Camel
waiting for the messages, make the tests much slower, because time is lost
on this waiting.
>>> I am assuming you use JUnit or Camel Test (based on JUnit). I also
>>> assume you are using the camel mock component based on (EasyMock). In
>>> this case, the mock component is expected to assert whether it received
>>> the message and whether the contents are as you expect them to be. If
>>> the mock component did not get an expected message, it is obvious you
>>> need to check a log to see what happened.
If you need to find out where in the route the message failed, you might
want to check out the Tracer component that will provide a trace of elements
of the route touched prior to failure.
http://camel.apache.org/tracer.html http://camel.apache.org/tracer.html
That said, you will still need to look at your logs... I doubt there is any
technology, product set, stack or other wise where you can figure out why a
message did not arrive by not looking at a log ;)
Hope this helps. While I am in no position to tell you how you should
proceed, this should hopefully address some of your technical misgivings
about Camel...
Cheers,
Ashwin...
-----
---------------------------------------------------------
Ashwin Karpe
Apache Camel Committer & Sr Principal Consultant
FUSESource (a Progress Software Corporation subsidiary)
http://fusesource.com
Blog: http://opensourceknowledge.blogspot.com
---------------------------------------------------------
--
View this message in context: http://camel.465427.n5.nabble.com/Question-on-the-proper-usage-of-Camel-tp5472181p5472699.html
Sent from the Camel - Users mailing list archive at Nabble.com.