You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by mgiammarco <mg...@gmail.com> on 2011/12/09 23:56:37 UTC
Newbie: is doable this application?
Hello,
I would like to use camel to realize this (apparently) simple web
application:
- a CONCURRENT MULTIUSER web application with a simple search page like
google;
- results MUST appears asynchronously as soon as they can;
- I need to query several databases (SQL,HTTP,WEBSERVICES,TELNET)
concurrently;
- The query in one database MUST be serialized.
I am worried regarding last point because obviously tomcat for each user
will create a new session and I do not know if camel can create a
"singleton" queue to serialize access to the db.
Thank you in advance for any help!
Mario
--
View this message in context: http://camel.465427.n5.nabble.com/Newbie-is-doable-this-application-tp5063252p5063252.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Newbie: is doable this application?
Posted by Christian Schneider <ch...@die-schneider.net>.
Am 10.12.2011 13:14, schrieb mgiammarco:
> First, thanks for your detailed reply!
>
> Christian Schneider wrote
>> you could use jms to achieve async behaviour and serialization.
>>
>> - You can limit the number of consumer threads to 1 and also the number
>> of threads for the executor thread pool to 1 to make sure each source
>> only processes one message at a time
>> - after the query you send the result back to a common response queue
>> - the web appplication listens on the response queue and prints each
>> response
>> One problem with this aproch is that you do not know exactly when all
>> responses are there. As long as the number of engines is fixed you can
>> count the responses though.
>>
>>
> Nice idea: separate frontend from backend and put a queue listener in front
> of backend.
> Some questions:
> - can I use amqp and not jms?
There is an amqp ccomponent in camel but I am not sure how good it is.
Most messaging systems provide a jms api though.
> - most sources can (and MUST) process more than one message at a time, can I
> give them more threads?
You can have different thread settings for each component. By putting a
queue after the topic for each component you can
even load balance to several servers for one component.
> - When I receive last reply I send a message "end of data" to web
> application so it knows I have finished.
As long as you know the number of components that should work. I would
consider using camel directly in the web app so you do not need an
additional layer.
Christian
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
Talend Application Integration Division http://www.talend.com
Re: Newbie: is doable this application?
Posted by mgiammarco <mg...@gmail.com>.
First, thanks for your detailed reply!
Christian Schneider wrote
>
> you could use jms to achieve async behaviour and serialization.
>
> - You can limit the number of consumer threads to 1 and also the number
> of threads for the executor thread pool to 1 to make sure each source
> only processes one message at a time
> - after the query you send the result back to a common response queue
> - the web appplication listens on the response queue and prints each
> response
> One problem with this aproch is that you do not know exactly when all
> responses are there. As long as the number of engines is fixed you can
> count the responses though.
>
>
Nice idea: separate frontend from backend and put a queue listener in front
of backend.
Some questions:
- can I use amqp and not jms?
- most sources can (and MUST) process more than one message at a time, can I
give them more threads?
- When I receive last reply I send a message "end of data" to web
application so it knows I have finished.
--
View this message in context: http://camel.465427.n5.nabble.com/Newbie-is-doable-this-application-tp5063252p5064221.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Newbie: is doable this application?
Posted by Christian Schneider <ch...@die-schneider.net>.
Hi Mario,
you could use jms to achieve async behaviour and serialization.
My design would look like this:
- one topic for the queries
- the web application sends the query to the topic
- you have one listener for each data source
- You can limit the number of consumer threads to 1 and also the number
of threads for the executor thread pool to 1 to make sure each source
only processes one message at a time
- after the query you send the result back to a common response queue
- the web appplication listens on the response queue and prints each
response
- to differentiate between the queries each query should have a query id
and the response should also contain the query id
So the routes could look like this:
For each search source you have:
from("jms:topic:mytopic").bean(new MysearchBean1()).to("jms:myreplyqueue");
class MySearchBean1() {
public Result search(Query query) {
...
}
}
Query and Result should be jaxb annotated classes. So if you have
camel-jaxb on the path camel converts from jms messages to the objects
automatically. (See also
http://www.liquid-reality.de/display/liquid/2011/10/19/Using+Camel+to+do+light+weight+messaging+over+any+protocol
)
In the web application you use a producerTemplate to send the query:
context.createProducerTemplate().sendBody("jms:topic:mytopic", query);
Alternatively you can use the aproach described in my blog article to
hide that behind a java interface.
A route in the web application receives the result:
from("jms:myreplyqueue").bean(new MyResponseHandler());
class MyresponseHandler {
public void handle(Response response);
}
So this class should figure out to which query the response belongs to
and display it in the correct session.
One problem with this aproch is that you do not know exactly when all
responses are there. As long as the number of engines is fixed you can
count the responses though.
Some advantages of this design are:
- You can easily add more engines and deploy them on the same or
different machines
- You can add more search sources on the fly
- Your java code will contain no references to camel so it is easy to
unit test independently from camel
Christian
Am 09.12.2011 23:56, schrieb mgiammarco:
> Hello,
> I would like to use camel to realize this (apparently) simple web
> application:
>
> - a CONCURRENT MULTIUSER web application with a simple search page like
> google;
> - results MUST appears asynchronously as soon as they can;
> - I need to query several databases (SQL,HTTP,WEBSERVICES,TELNET)
> concurrently;
> - The query in one database MUST be serialized.
>
> I am worried regarding last point because obviously tomcat for each user
> will create a new session and I do not know if camel can create a
> "singleton" queue to serialize access to the db.
>
> Thank you in advance for any help!
>
> Mario
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Newbie-is-doable-this-application-tp5063252p5063252.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
Talend Application Integration Division http://www.talend.com
Re: Newbie: is doable this application?
Posted by mgiammarco <mg...@gmail.com>.
Preben.Asmussen wrote
>
>
> Maybe you could create a Lucene or Solr index from the resources, and then
> use that for your user search.
> See http://lucene.apache.org/java/docs/index.html
>
>
>
Thank you for suggestion, unfortunately some of this data bases are "pay per
use" so at most I can put a memcache to cache past queries.
--
View this message in context: http://camel.465427.n5.nabble.com/Newbie-is-doable-this-application-tp5063252p5075610.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Newbie: is doable this application?
Posted by "Preben.Asmussen" <pr...@dr.dk>.
As I understand it you need to query several resources for a searchresult
asynchronously. My guess is that it would be hard to have a responsive
Google like search traveling several resources in a search.
Maybe you could create a Lucene or Solr index from the resources, and then
use that for your user search.
See http://lucene.apache.org/java/docs/index.html
Preben
--
View this message in context: http://camel.465427.n5.nabble.com/Newbie-is-doable-this-application-tp5063252p5065536.html
Sent from the Camel - Users mailing list archive at Nabble.com.