Re: Would appreciate advice on the best way to tackle a problem

[Christian Schneider <ch...@die-schneider.net>]

Hi Alph,

your problem seems to really cry for jms. So if you do not absolutely 
have to use soap/http for your webservice I advise to do the following.

I am not exactly sure how to do the database polling but you ssound as 
if you know how to do it. So I absstract this away. As your service is 
completely asynchronous you can also nicely skip the soap part and 
instead I would send a jaxb serialized data class.

from("db").to("bean:transfomer").to("jms")

from("jms").transactional().to("bean:servicebean")

So the idea is to poll the database somehow then use a bean to transform 
to your data class. If the classs is jaxb annotated it will be 
serialized to xml before calling jms. You only have to have the 
camel-jaxb jar on the classpath.

So the bean transformer would be defined in spring and has a method like 
this:

DataClass transform(WhateverTheFatabasePollingProduces source) {
};

On the service sidde you have a similar bean with:

void service(DataClasss) {
};

Now to achieve the behaviour with the guaranteed deliver and error 
handling you require you can set the queue options so it does several 
redelivery attempts and then move the message to an error queue if that 
fails.

This may sound a bit unusual if you are not experienced with jms but 
this iss exactly the case jms is best in.

Christian


Am 19.04.2011 03:32, schrieb alpheratz:
> Looking for advice on the best way to tackle the following...
>
> Need a process to:
> + poll a database table for the arrival of a new record
> + transform the record to the XML appropriate for a document-literal SOAP
> webservice
> + send to the webservice
>
> All this is relatively easy; here are the issues:
> + the webservice will respond with an immediate communication status
> (ok/fail)
> + the service will also respond at some later time with an asynchronous
> processing status (ok/fail)
> + outgoing message (n + 1) should only be sent on receipt of the processing
> status response for message (n); polling may be paused waiting for this or
> it may continue (requiring messages to be buffered, presumably...this is the
> easier alternative maybe?)
> + the incoming async response message may never arrive, of course, so
> timeout handling will be required
> + the communication status should ideally allow one to distinguish between
> transient comms failures and permanent ones. In the case of transient one
> should retry while permanent errors should result in the message being
> logged
>
> I'm not a COMPLETE neophyte with Camel but I'd appreciate any expert advice
> on the best way to structure a solution. Pointers to examples/blogs/etc.
> would be great!
>
> Thoughts/suggestions gratefully accepted.
>
> Thanks in advance.
>
> Cheers,
>
> Alph
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Would-appreciate-advice-on-the-best-way-to-tackle-a-problem-tp4312202p4312202.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>

-- 
----
http://www.liquid-reality.de

Re: Would appreciate advice on the best way to tackle a problem

[Christian Schneider <ch...@die-schneider.net>]

I think a good solution may be to do the source side like I suggested. 
So you go from database to jms. Then you can use a route that listens on 
jms with one thread. This will serialize the
processing. From there you can call the destination application 
synchronously. The only hard thing may be to feed things into jms in the 
right order. I think the best way may be to store one master and all 
connected detail records for the destination in one jms message for each 
master record.

Christian


Am 20.04.2011 01:12, schrieb alpheratz:
> Christian Schneider-3 wrote:
>> this would be quite difficult to achieve...
>>
> Yes. My colleagues are proposing a heath-robinson-ish mass of stored
> procedures and triggers but I would like to avoid spattering rubbish across
> our database if I can.

-- 
----
http://www.liquid-reality.de

Re: Would appreciate advice on the best way to tackle a problem

[alpheratz <ca...@transentia.com.au>]

hadrian wrote:
> 
> did you consider camel-bam [1]?
> (The "b.starts().after(a.completes())" looks like what you're looking
> for).
> 

I have never heard of this...

It looks interesting, but I'm not sure that it is really appropriate...it
seems to be encoding a fixed relationship between messages/entities. In my
case, I have variable (unknown, including 0) number of 'b' messages that may
follow an 'a'. Thus I have an less precisely defined relationship...

Still, I will read up more.

Thanks.

--
View this message in context: http://camel.465427.n5.nabble.com/Would-appreciate-advice-on-the-best-way-to-tackle-a-problem-tp4312202p4314925.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Would appreciate advice on the best way to tackle a problem

[Hadrian Zbarcea <hz...@gmail.com>]

Hi,

Christian is right, looks like jms should definitely be part of the solution. For the other part, did you consider camel-bam [1]?
(The "b.starts().after(a.completes())" looks like what you're looking for).
 
Cheers,
Hadrian

[1] http://camel.apache.org/bam.html

On Apr 19, 2011, at 7:12 PM, alpheratz wrote:

> 
> Christian Schneider-3 wrote:
>> 
>> this would be quite difficult to achieve...
>> 
> 
> Yes. My colleagues are proposing a heath-robinson-ish mass of stored
> procedures and triggers but I would like to avoid spattering rubbish across
> our database if I can.
> 
> 
> Christian Schneider-3 wrote:
>> 
>> The question is do you really need to delay the message? 
>> 
> 
> Sadly, yes. 
> 
> I am trying to integrate with a 3rd party application that places this
> "Don't even think about sending me a message until I tell you that I have
> processed the previous one" requirement on me. 
> 
> Not good, but there it is.
> 
> It is not quite as bad as it sounds: there can be multiple message steams
> active at once but within a stream everything has to be strictly serialized;
> it's a master-detail situation. I send a master mesage and then can send
> subsequent detail records. These detail records have to be fully processed
> one at a time.
> 
> So it is not just about order but also about flow control, as you point out
> (thanks). I guess that this means that I can't just manipulate priorities as
> I first thought but need to play with the release strategies of a queue.
> 
> --
> View this message in context: http://camel.465427.n5.nabble.com/Would-appreciate-advice-on-the-best-way-to-tackle-a-problem-tp4312202p4314564.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Would appreciate advice on the best way to tackle a problem

[alpheratz <ca...@transentia.com.au>]

Christian Schneider-3 wrote:
> 
> this would be quite difficult to achieve...
> 

Yes. My colleagues are proposing a heath-robinson-ish mass of stored
procedures and triggers but I would like to avoid spattering rubbish across
our database if I can.


Christian Schneider-3 wrote:
> 
> The question is do you really need to delay the message? 
> 

Sadly, yes. 

I am trying to integrate with a 3rd party application that places this
"Don't even think about sending me a message until I tell you that I have
processed the previous one" requirement on me. 

Not good, but there it is.

It is not quite as bad as it sounds: there can be multiple message steams
active at once but within a stream everything has to be strictly serialized;
it's a master-detail situation. I send a master mesage and then can send
subsequent detail records. These detail records have to be fully processed
one at a time.

So it is not just about order but also about flow control, as you point out
(thanks). I guess that this means that I can't just manipulate priorities as
I first thought but need to play with the release strategies of a queue.

--
View this message in context: http://camel.465427.n5.nabble.com/Would-appreciate-advice-on-the-best-way-to-tackle-a-problem-tp4312202p4314564.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Would appreciate advice on the best way to tackle a problem

[Christian Schneider <ch...@sopera.com>]

Hi Alph,

this would be quite difficult to achieve as you would then have to 
manage the flow by hand again.

The question is do you really need to delay the message? The jms broker 
can guarantee that it delivers the message so I would just send all 
messages to the broker and then let receiver handle them one after the 
other. I think you can guarantee that the order of messages is kept.

Christian


Am 19.04.2011 12:25, schrieb alpheratz:
> Thanks for taking the time to respond to me. Much appreciated.
>
> Your suggestion makes good sense to me.
>
> I am mostly worrying about how to achieve the "delay message (n + 1) until
> message (n) is acknowledged" part. I assume that one would have to have a
> priority queue of some sort, with the message priorities readjusted
> according to the incoming asynchronous acknowledgements. Not sure how to
> achieve this...certainly not in a clean way.
>
> Cheers,
>
> Alph
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Would-appreciate-advice-on-the-best-way-to-tackle-a-problem-tp4312202p4312954.html
> Sent from the Camel - Users mailing list archive at Nabble.com.


-- 
Christian Schneider
CXF and Camel Architect
SOPERA - The Application Integration Division of Talend
http://www.talend.com

Re: Would appreciate advice on the best way to tackle a problem

[alpheratz <ca...@transentia.com.au>]

Thanks for taking the time to respond to me. Much appreciated.

Your suggestion makes good sense to me. 

I am mostly worrying about how to achieve the "delay message (n + 1) until
message (n) is acknowledged" part. I assume that one would have to have a
priority queue of some sort, with the message priorities readjusted
according to the incoming asynchronous acknowledgements. Not sure how to
achieve this...certainly not in a clean way.

Cheers,

Alph



--
View this message in context: http://camel.465427.n5.nabble.com/Would-appreciate-advice-on-the-best-way-to-tackle-a-problem-tp4312202p4312954.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Would appreciate advice on the best way to tackle a problem

[Christian Schneider <ch...@die-schneider.net>]

What I forgot to write:

from("jms") should be from("jms:queuename").

Some small explanation how your servicebean behaves in this. If the 
method call succeeds the message is automatically committed and so 
removed from the queue. If the method call throws an exception the 
message is automatically rolled back and will be redelivered. If the 
server does down while processing the message the message is also retried.

You can even have several services with a route like this and they will 
do failover and loadbalancing without further code.

Christian


Am 19.04.2011 08:26, schrieb Christian Schneider:
> Hi Alph,
>
> your problem seems to really cry for jms. So if you do not absolutely 
> have to use soap/http for your webservice I advise to do the following.
>
> I am not exactly sure how to do the database polling but you ssound as 
> if you know how to do it. So I absstract this away. As your service is 
> completely asynchronous you can also nicely skip the soap part and 
> instead I would send a jaxb serialized data class.
>
> from("db").to("bean:transfomer").to("jms")
>
> from("jms").transactional().to("bean:servicebean")
>
> So the idea is to poll the database somehow then use a bean to 
> transform to your data class. If the classs is jaxb annotated it will 
> be serialized to xml before calling jms. You only have to have the 
> camel-jaxb jar on the classpath.
>
> So the bean transformer would be defined in spring and has a method 
> like this:
>
> DataClass transform(WhateverTheFatabasePollingProduces source) {
> };
>
> On the service sidde you have a similar bean with:
>
> void service(DataClasss) {
> };
>
> Now to achieve the behaviour with the guaranteed deliver and error 
> handling you require you can set the queue options so it does several 
> redelivery attempts and then move the message to an error queue if 
> that fails.
>
> This may sound a bit unusual if you are not experienced with jms but 
> this iss exactly the case jms is best in.
>
> Christian
>
>
> Am 19.04.2011 03:32, schrieb alpheratz:
>> Looking for advice on the best way to tackle the following...
>>
>> Need a process to:
>> + poll a database table for the arrival of a new record
>> + transform the record to the XML appropriate for a document-literal 
>> SOAP
>> webservice
>> + send to the webservice
>>
>> All this is relatively easy; here are the issues:
>> + the webservice will respond with an immediate communication status
>> (ok/fail)
>> + the service will also respond at some later time with an asynchronous
>> processing status (ok/fail)
>> + outgoing message (n + 1) should only be sent on receipt of the 
>> processing
>> status response for message (n); polling may be paused waiting for 
>> this or
>> it may continue (requiring messages to be buffered, presumably...this 
>> is the
>> easier alternative maybe?)
>> + the incoming async response message may never arrive, of course, so
>> timeout handling will be required
>> + the communication status should ideally allow one to distinguish 
>> between
>> transient comms failures and permanent ones. In the case of transient 
>> one
>> should retry while permanent errors should result in the message being
>> logged
>>
>> I'm not a COMPLETE neophyte with Camel but I'd appreciate any expert 
>> advice
>> on the best way to structure a solution. Pointers to examples/blogs/etc.
>> would be great!
>>
>> Thoughts/suggestions gratefully accepted.
>>
>> Thanks in advance.
>>
>> Cheers,
>>
>> Alph
>>
>> -- 
>> View this message in context: 
>> http://camel.465427.n5.nabble.com/Would-appreciate-advice-on-the-best-way-to-tackle-a-problem-tp4312202p4312202.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>

-- 
----
http://www.liquid-reality.de