You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Neil Rutherford <ne...@xor.za.net> on 2009/03/11 07:53:58 UTC
trying to implement rendezvous
hi,
I've been tasked with doing a proof of concept project using Camel, the goal
is to implement a message switching server.
The switch needs to do the following:
- receive XML messages via TCP socket
- these messages are made up of smaller messages, so the switch will need to
split them into seperate messages
- there are 2 types of smaller messages in these submessages: 1 control
message and n number of service messages
- the control message needs to be submitted first and if it is processed
successfully the service messages can be submitted/processed (in parallel?)
- the responses from all the submessages need to be aggregated into a single
response message to send back out the open TCP socket
I've tried implementing this rendezvous process using custom processors and
custom components with very little success. The rendezvous processing will
need to be done asynchronously as there will be many messages being
processed at the same time and the time to process submessages is out of my
hands (they go on to other servers via TCP).
I was thinking of persisting service messages to DB, continuing on to
process control message and then pass response back to rendezvous
processor/component, have it load service messages from DB and process them.
socket -> rendezvous -> content based router -> submessage processor ->
rendezvous -> ...
I'm probably not being very clear, so feel free to ask specific questions.
thanks,
Neil
P.S. Rendezvous may also be the wrong word, its just what we call it here.
--
View this message in context: http://www.nabble.com/trying-to-implement-rendezvous-tp22449660p22449660.html
Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
Re: trying to implement rendezvous
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Mar 11, 2009 at 10:55 AM, Neil Rutherford <nr...@gmail.com> wrote:
> With this approach:
>
> from(tcp).process(controlMessage).to("direct:submsg");
> from("direct:submsg").split().parallelProcessing().to(client socket).end();
>
> How does the threading work?
>
> Will a new thread be spawned for each new request into the tcp
> endpoint and will that thread then spawn a new thread for each
> submessage @ the parallelprocessing() part and will each of those
> threads block while waiting for a response from the client socket?
>
> So for example, if 10 requests came in at the same second:
>
> - 10 threads will be spawned to process the route
> - each message has 5 service messages and so each processing thread
> will now spawn 5 more threads to process the service messages
> - all these threads will block while waiting for a response or socket
> timeout (timeout for client socket is 30 sec)
Camels uses the JDK ExecutorService concurrency framework introduced in Java5.
By default there is a pool of up till 10 concurrent threads. Its the
JDK that handles the scheduling and all that stuff.
You can set you own configuration of this on the component with the
setExecutorService.
And some of the EIP patterns in Camel have individual configuration.
In fact the splitter also have in 1.6.0+
http://camel.apache.org/splitter.html
>
> thanks,
>
> Neil
>
> On 11/03/2009, Claus Ibsen <cl...@gmail.com> wrote:
>
>> Since the control message is the key, you should wait with the
>> splitting until the control message has been submitted and processed
>> succesful.
>>
>> // an exception catcher to return a failed MSG to the TCP (*)
>> onException(MyProcessException.class).handled(true).process(setFailedResponseMsg);
>> // and then set a failed msg in the OUT body
>>
>> // a simple route for the control.
>> from(tcp).process(controlMessage).to("direct:submsg");
>>
>> // and the splitter has a build in aggregator, so we can do it
>> something like this
>> // See more at: http://camel.apache.org/splitter.html, there is a
>> sample at the bottom at split, aggregate and return a combined
>> response
>> from(direct:submsg).split(xpath stuff, aggregation
>> strategy).parallelProcessing().to(processEachLine).end();
>>
>>
>> Ad *
>> You can use tryBlock(), handle() instead as a kind of try .. catch
>> code as well. But then you can not do multiple routes (eg multiple
>> from)
>> So its something like this.
>>
>> tryBlock();
>> from(tcp).process(controlMessage).
>> .split(xpath stuff, aggregation
>> strategy).parallelProcessing().to(processEachLine).end();
>> .handle(Exception.class)
>> .process(setFailedResponseMsg);
>> .end();
>>
>> Note I wrote this directly in gmail, so no code completion and you
>> might need to fiddle with it to get the syntax correct and it to work.
>>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Re: trying to implement rendezvous
Posted by Neil Rutherford <nr...@gmail.com>.
With this approach:
from(tcp).process(controlMessage).to("direct:submsg");
from("direct:submsg").split().parallelProcessing().to(client socket).end();
How does the threading work?
Will a new thread be spawned for each new request into the tcp
endpoint and will that thread then spawn a new thread for each
submessage @ the parallelprocessing() part and will each of those
threads block while waiting for a response from the client socket?
So for example, if 10 requests came in at the same second:
- 10 threads will be spawned to process the route
- each message has 5 service messages and so each processing thread
will now spawn 5 more threads to process the service messages
- all these threads will block while waiting for a response or socket
timeout (timeout for client socket is 30 sec)
thanks,
Neil
On 11/03/2009, Claus Ibsen <cl...@gmail.com> wrote:
> Since the control message is the key, you should wait with the
> splitting until the control message has been submitted and processed
> succesful.
>
> // an exception catcher to return a failed MSG to the TCP (*)
> onException(MyProcessException.class).handled(true).process(setFailedResponseMsg);
> // and then set a failed msg in the OUT body
>
> // a simple route for the control.
> from(tcp).process(controlMessage).to("direct:submsg");
>
> // and the splitter has a build in aggregator, so we can do it
> something like this
> // See more at: http://camel.apache.org/splitter.html, there is a
> sample at the bottom at split, aggregate and return a combined
> response
> from(direct:submsg).split(xpath stuff, aggregation
> strategy).parallelProcessing().to(processEachLine).end();
>
>
> Ad *
> You can use tryBlock(), handle() instead as a kind of try .. catch
> code as well. But then you can not do multiple routes (eg multiple
> from)
> So its something like this.
>
> tryBlock();
> from(tcp).process(controlMessage).
> .split(xpath stuff, aggregation
> strategy).parallelProcessing().to(processEachLine).end();
> .handle(Exception.class)
> .process(setFailedResponseMsg);
> .end();
>
> Note I wrote this directly in gmail, so no code completion and you
> might need to fiddle with it to get the syntax correct and it to work.
>
Re: trying to implement rendezvous
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Mar 11, 2009 at 7:53 AM, Neil Rutherford <ne...@xor.za.net> wrote:
>
> hi,
>
> I've been tasked with doing a proof of concept project using Camel, the goal
> is to implement a message switching server.
>
> The switch needs to do the following:
>
> - receive XML messages via TCP socket
> - these messages are made up of smaller messages, so the switch will need to
> split them into seperate messages
> - there are 2 types of smaller messages in these submessages: 1 control
> message and n number of service messages
> - the control message needs to be submitted first and if it is processed
> successfully the service messages can be submitted/processed (in parallel?)
> - the responses from all the submessages need to be aggregated into a single
> response message to send back out the open TCP socket
Since the control message is the key, you should wait with the
splitting until the control message has been submitted and processed
succesful.
// an exception catcher to return a failed MSG to the TCP (*)
onException(MyProcessException.class).handled(true).process(setFailedResponseMsg);
// and then set a failed msg in the OUT body
// a simple route for the control.
from(tcp).process(controlMessage).to("direct:submsg");
// and the splitter has a build in aggregator, so we can do it
something like this
// See more at: http://camel.apache.org/splitter.html, there is a
sample at the bottom at split, aggregate and return a combined
response
from(direct:submsg).split(xpath stuff, aggregation
strategy).parallelProcessing().to(processEachLine).end();
Ad *
You can use tryBlock(), handle() instead as a kind of try .. catch
code as well. But then you can not do multiple routes (eg multiple
from)
So its something like this.
tryBlock();
from(tcp).process(controlMessage).
.split(xpath stuff, aggregation
strategy).parallelProcessing().to(processEachLine).end();
.handle(Exception.class)
.process(setFailedResponseMsg);
.end();
Note I wrote this directly in gmail, so no code completion and you
might need to fiddle with it to get the syntax correct and it to work.
>
> I've tried implementing this rendezvous process using custom processors and
> custom components with very little success. The rendezvous processing will
> need to be done asynchronously as there will be many messages being
> processed at the same time and the time to process submessages is out of my
> hands (they go on to other servers via TCP).
>
> I was thinking of persisting service messages to DB, continuing on to
> process control message and then pass response back to rendezvous
> processor/component, have it load service messages from DB and process them.
>
> socket -> rendezvous -> content based router -> submessage processor ->
> rendezvous -> ...
>
> I'm probably not being very clear, so feel free to ask specific questions.
>
> thanks,
>
> Neil
>
> P.S. Rendezvous may also be the wrong word, its just what we call it here.
>
> --
> View this message in context: http://www.nabble.com/trying-to-implement-rendezvous-tp22449660p22449660.html
> Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/