You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by mk <mk...@gmail.com> on 2012/08/09 12:36:22 UTC
Camel usage in Online usecases
Hi,
In our product, we have the following requirements:
1) In the data ingest side, we will have around 5 million records/sec
traffic. These CDR records could come to us in various transports including
file, JMS, XML over HTTP etc. This data has to be received and written into
HBase
2) In the output side, we have an online module connected to customer care
applications. Here again, we need to accept queries over various transports
like HTTP, JMS, bulk file etc. The system needs to parse the query,
implement business logic and retrieve data from HBase, send the response to
the online query.
Since the product involved a lot of transport support (like HTTP, JMS,
future expandable to TCP, UDP streams), Im considering bringing in Apache
Camel.
Does Camel suit these kind of online usecases where performance and latency
are important parameters? Does the Camel layer significantly impact latency?
It would be very helpful if someone can:
1) Throw some light on the performance/latency issues in Camel, if any
2) Point to existing production examples that use Camel for similar online
usecases
3) Point to other resources which could throw more light on this subject
Thanks & Regards
MK
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-usage-in-Online-usecases-tp5717058.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel usage in Online usecases
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Aug 16, 2012 at 8:53 AM, Karthikeyan Muthukumarasamy
<mk...@gmail.com> wrote:
> Hi Michael,
> Thanks for the reply. Sure, I will look into the page and will also share
> my benchmarks with the community.
> Meanwhile, can you or someone else, please point me to some material on the
> internet which mentions usage of Camel in online, low-latency applications?
The CamelOne conferences in 2011 and 2012 had speakers from various
companies talking about their usage of Camel.
Or related frameworks.
http://fusesource.com/apache-camel-conference-2012/
Some of the commercial vendors such as FuseSource have use-cases /
whitepapers etc about usage of Camel at large enterprises etc.
http://fusesource.com/community/fuse-customers/
And there may be some links to blogs / articles etc about usage of
Camel at various companies, from this link collection
http://camel.apache.org/articles
> Thanks
> MK
>
> On Fri, Aug 10, 2012 at 12:20 PM, michal.warecki [via Camel] <
> ml-node+s465427n5717114h83@n5.nabble.com> wrote:
>
>> Look at S. England-Sullivan's post on
>> http://camel.465427.n5.nabble.com/fyi-SI-td5716049.html .
>> There are performance tests with Camel+JMS and Camel+Simple JMS. With
>> proper configuration Camel+JMS overhead (JMS is quite heavy protocol) is
>> from 1367ms to 424ms for 1000 msgs, so 100 msgs/s is not a problem.
>>
>> Camel provides the ability to create mediation routes in minutes, so I
>> think you should just create a benchmark tests. Do not forget to share with
>> us the results :-)
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://camel.465427.n5.nabble.com/Camel-usage-in-Online-usecases-tp5717058p5717114.html
>> To unsubscribe from Camel usage in Online usecases, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5717058&code=bWthcnRoaWsuaGVyZUBnbWFpbC5jb218NTcxNzA1OHwtMzc4NDc0NzY2>
>> .
>> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: Camel usage in Online usecases
Posted by Karthikeyan Muthukumarasamy <mk...@gmail.com>.
Hi Michael,
Thanks for the reply. Sure, I will look into the page and will also share
my benchmarks with the community.
Meanwhile, can you or someone else, please point me to some material on the
internet which mentions usage of Camel in online, low-latency applications?
Thanks
MK
On Fri, Aug 10, 2012 at 12:20 PM, michal.warecki [via Camel] <
ml-node+s465427n5717114h83@n5.nabble.com> wrote:
> Look at S. England-Sullivan's post on
> http://camel.465427.n5.nabble.com/fyi-SI-td5716049.html .
> There are performance tests with Camel+JMS and Camel+Simple JMS. With
> proper configuration Camel+JMS overhead (JMS is quite heavy protocol) is
> from 1367ms to 424ms for 1000 msgs, so 100 msgs/s is not a problem.
>
> Camel provides the ability to create mediation routes in minutes, so I
> think you should just create a benchmark tests. Do not forget to share with
> us the results :-)
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://camel.465427.n5.nabble.com/Camel-usage-in-Online-usecases-tp5717058p5717114.html
> To unsubscribe from Camel usage in Online usecases, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5717058&code=bWthcnRoaWsuaGVyZUBnbWFpbC5jb218NTcxNzA1OHwtMzc4NDc0NzY2>
> .
> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
Re: Camel usage in Online usecases
Posted by "michal.warecki" <mi...@gmail.com>.
Look at S. England-Sullivan's post on
http://camel.465427.n5.nabble.com/fyi-SI-td5716049.html .
There are performance tests with Camel+JMS and Camel+Simple JMS. With proper
configuration Camel+JMS overhead (JMS is quite heavy protocol) is from
1367ms to 424ms for 1000 msgs, so 100 msgs/s is not a problem.
Camel provides the ability to create mediation routes in minutes, so I think
you should just create a benchmark tests. Do not forget to share with us the
results :-)
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-usage-in-Online-usecases-tp5717058p5717114.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel usage in Online usecases
Posted by Karthikeyan Muthukumarasamy <mk...@gmail.com>.
Hi Michal,
Thanks for your response.
>>> Do you need low latency? What are your SLA for avg and max response
time?
100 queries/sec with max. latency of 1 sec
>>> For what perposes you plan to use Camel? To unify transport layer
configuration? Do you plan to rewrite your integration architecture to
direct use of database from other transport layer? If so I think Camel is a
good choise to consider.
The main 2 purposes of bringing in camel are as follows:
- Avail advantage of the multi-transport (eg. SOAP, HTTP, TCP, JMS, File
etc) support so we reduce writing transport infrastructure code
- Avail the integration patterns enforced by CAMEL to ensure that data
received from any transport is converted into a common object format and
then sent to the business logic bean, whcih would implement the business
logic in a transport-agnostic fashion (similar to the Rider Auto Parts
example in Camel In Action book)
Now, the question is:
For the latency & traffic requirements of the product stated above, is it
advantageous to use Camel for the above two purposes?
Thanks
MK
On Fri, Aug 10, 2012 at 2:13 AM, michal.warecki [via Camel] <
ml-node+s465427n5717092h1@n5.nabble.com> wrote:
> It is about 60 msg per sec in average so it is not the problem. Like other
> frameworks Camel has some overhead. Do you need low latency? What are your
> SLA for avg and max response time? For what perposes you plan to use Camel?
> To unify transport layer configuration? Do you plan to rewrite your
> integration architecture to direct use of database from other transport
> layer? If so I think Camel is a good choise to consider.
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://camel.465427.n5.nabble.com/Camel-usage-in-Online-usecases-tp5717058p5717092.html
> To unsubscribe from Camel usage in Online usecases, click here<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5717058&code=bWthcnRoaWsuaGVyZUBnbWFpbC5jb218NTcxNzA1OHwtMzc4NDc0NzY2>
> .
> NAML<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
Re: Camel usage in Online usecases
Posted by "michal.warecki" <mi...@gmail.com>.
It is about 60 msg per sec in average so it is not the problem. Like other
frameworks Camel has some overhead. Do you need low latency? What are your
SLA for avg and max response time? For what perposes you plan to use Camel?
To unify transport layer configuration? Do you plan to rewrite your
integration architecture to direct use of database from other transport
layer? If so I think Camel is a good choise to consider.
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-usage-in-Online-usecases-tp5717058p5717092.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel usage in Online usecases
Posted by Karthikeyan Muthukumarasamy <mk...@gmail.com>.
Hi,
Oops! Sorry that was a typo.
Its 5 million records/day and not /sec.
MK
On Thu, Aug 9, 2012 at 4:29 PM, Joe San <co...@gmail.com> wrote:
> *In the data ingest side, we will have around 5 million records/sec
> traffic*
>
> Are you serious with your above statement? How big is that data record. Are
> you using Super Computing? I've never seen a system that can handle this.
>
> Regards,
> Jothi
>
> On Thu, Aug 9, 2012 at 12:36 PM, mk <mk...@gmail.com> wrote:
>
> > Hi,
> > In our product, we have the following requirements:
> > 1) In the data ingest side, we will have around 5 million records/sec
> > traffic. These CDR records could come to us in various transports
> including
> > file, JMS, XML over HTTP etc. This data has to be received and written
> into
> > HBase
> > 2) In the output side, we have an online module connected to customer
> care
> > applications. Here again, we need to accept queries over various
> transports
> > like HTTP, JMS, bulk file etc. The system needs to parse the query,
> > implement business logic and retrieve data from HBase, send the response
> to
> > the online query.
> > Since the product involved a lot of transport support (like HTTP, JMS,
> > future expandable to TCP, UDP streams), Im considering bringing in Apache
> > Camel.
> > Does Camel suit these kind of online usecases where performance and
> latency
> > are important parameters? Does the Camel layer significantly impact
> > latency?
> > It would be very helpful if someone can:
> > 1) Throw some light on the performance/latency issues in Camel, if any
> > 2) Point to existing production examples that use Camel for similar
> online
> > usecases
> > 3) Point to other resources which could throw more light on this subject
> > Thanks & Regards
> > MK
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://camel.465427.n5.nabble.com/Camel-usage-in-Online-usecases-tp5717058.html
> > Sent from the Camel - Users mailing list archive at Nabble.com.
> >
>
Re: Camel usage in Online usecases
Posted by Joe San <co...@gmail.com>.
*In the data ingest side, we will have around 5 million records/sec traffic*
Are you serious with your above statement? How big is that data record. Are
you using Super Computing? I've never seen a system that can handle this.
Regards,
Jothi
On Thu, Aug 9, 2012 at 12:36 PM, mk <mk...@gmail.com> wrote:
> Hi,
> In our product, we have the following requirements:
> 1) In the data ingest side, we will have around 5 million records/sec
> traffic. These CDR records could come to us in various transports including
> file, JMS, XML over HTTP etc. This data has to be received and written into
> HBase
> 2) In the output side, we have an online module connected to customer care
> applications. Here again, we need to accept queries over various transports
> like HTTP, JMS, bulk file etc. The system needs to parse the query,
> implement business logic and retrieve data from HBase, send the response to
> the online query.
> Since the product involved a lot of transport support (like HTTP, JMS,
> future expandable to TCP, UDP streams), Im considering bringing in Apache
> Camel.
> Does Camel suit these kind of online usecases where performance and latency
> are important parameters? Does the Camel layer significantly impact
> latency?
> It would be very helpful if someone can:
> 1) Throw some light on the performance/latency issues in Camel, if any
> 2) Point to existing production examples that use Camel for similar online
> usecases
> 3) Point to other resources which could throw more light on this subject
> Thanks & Regards
> MK
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Camel-usage-in-Online-usecases-tp5717058.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>