You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by erstanl <er...@gmail.com> on 2012/12/06 22:55:18 UTC
transactions that span multiple routes
Here is the use case.
1) Consume message from ActiveMQ queue
2) Update database with data derived from step 1
3) Download rather large data from database
4) Process data from step 3
5) Update database with results from step 4
These routes represent that case (I think):
Steps 1, 2
<route id="consumeReadyQueue">
<from uri="q:queue:queue/batch/ready" />
<transacted/>
<to uri="bean:batchListener?method=init" />
<to uri="direct:processBatch" />
</route>
Steps 3, 4, 5
<route id="processBatch">
<from uri="direct:processBatch" />
<to uri="bean:batchProcessor?method=process" />
</route>
The problem is, I only want steps 1 and 2 (route: consumeReadyQueue) to
participate in the XA transaction. From what I have gleaned,
consumeReadyQueue forces processBatch to participate in the transaction.
This is not good for my case as processBatch can take up to 30 seconds. Is
there a way to force the commit to happen before processBatch is running?
Maybe I am approaching this wrong. What are my possible options?
--
View this message in context: http://camel.465427.n5.nabble.com/transactions-that-span-multiple-routes-tp5723735.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: transactions that span multiple routes
Posted by erstanl <er...@gmail.com>.
Thanks for the response Christian, and sorry for my delay.
Christian Mueller wrote
> I do not have a solution at present, but I'm wondering what happens if you
> commit the exchange (than the message is deleted from the queue and the
> database is updated) and than your route fails?
> Also as far as I know, the default transaction timeout for SPTM is 120
> seconds. May be your 30 seconds are ok for one transaction?
If the message is
(1) deleted from the queue
(2) the database is updated
(3) and the route fails
This is perfectly fine. I can package up the exception and get that back to
the client.
It would be cool if we could demarcate the transaction differently. Instead
of:
<route>
<from uri="q:queue:queue/batch/ready" />
<transacted/>
<to uri="bean:batchListener?method=init" />
<to uri="seda:queue.process" />
</route>
<route>
<from uri="seda:queue.process" />
<to uri="bean:batchProcessor?method=process" />
</route>
one could do something like this:
<route>
<from uri="q:queue:queue/batch/ready" />
<transacted>
<to uri="bean:batchListener?method=init" />
</transacted>
<to uri="bean:batchProcessor?method=process" />
</route>
Is there another way for me to setup these routes to achieve my original use
case? I really would like to keep using camel, but this may force me to go
another route. Thanks for your help.
--
View this message in context: http://camel.465427.n5.nabble.com/transactions-that-span-multiple-routes-tp5723735p5734578.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: transactions that span multiple routes
Posted by Christian Müller <ch...@gmail.com>.
I do not have a solution at present, but I'm wondering what happens if you
commit the exchange (than the message is deleted from the queue and the
database is updated) and than your route fails?
Also as far as I know, the default transaction timeout for SPTM is 120
seconds. May be your 30 seconds are ok for one transaction?
Best,
Christian
On Wed, Jan 9, 2013 at 6:51 PM, erstanl <er...@gmail.com> wrote:
> Willem, thanks for the idea. Here is a resulting issue I have run into. The
> 1st route takes a message from queue/batch/ready, it throws it into the
> seda
> queue. It basically takes every message it can and stuffs it into the seda
> queue as fast as it can. The second route does process them as I would
> expect, but I cannot allow the first route to consume more than one at a
> time. There are many more machines running this same route that will not
> have a chance to get any messages.
>
> <route>
> <from uri="q:queue:queue/batch/ready" />
> <transacted/>
> <to uri="bean:batchListener?method=init" />
> <to uri="seda:queue.process" />
> </route>
>
> <route>
> <from uri="seda:queue.process" />
> <to uri="bean:batchProcessor?method=process" />
> </route>
>
> So I came up with this. It fulfills the ability to consume only as many as
> needed to process, but I don't think its handling the transaction boundary
> well because of the "blockWhenFull" attribute. It appears that the blocking
> on the first route is holding the transaction open until it is able to
> commit the message to the seda queue.
>
> <route>
> <from uri="q:queue:queue/batch/ready" />
> <transacted/>
> <to uri="bean:batchListener?method=init" />
> <to
>
> uri="seda:queue.process?size=2&blockWhenFull=true&concurrentConsumers=1"
> />
> </route>
>
> <route>
> <from
>
> uri="seda:queue.process?size=2&blockWhenFull=true&concurrentConsumers=1"
> />
> <to uri="bean:batchProcessor?method=process" />
> </route>
>
> I understand that the seda component is asynchronous, and that we are
> essentially using it to clip off the transaction at the route boundary. But
> that has its obvious downside of draining the actvemq queue in rapid
> fashion
> and queuing them in seda. It is almost as if I need to us a vm component
> instead of seda, but vm will participate in the transaction, which is the
> original problem. I have looked into using a throttle on the first route,
> but the data size being transferred is very inconsistent, and I cant rely
> on
> the throttle to feed the second route appropriately.
>
> I appreciate your help. If you could please let me know how I might be able
> to use camel to achieve what I have mentioned above.
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/transactions-that-span-multiple-routes-tp5723735p5725194.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Re: transactions that span multiple routes
Posted by erstanl <er...@gmail.com>.
Willem, thanks for the idea. Here is a resulting issue I have run into. The
1st route takes a message from queue/batch/ready, it throws it into the seda
queue. It basically takes every message it can and stuffs it into the seda
queue as fast as it can. The second route does process them as I would
expect, but I cannot allow the first route to consume more than one at a
time. There are many more machines running this same route that will not
have a chance to get any messages.
<route>
<from uri="q:queue:queue/batch/ready" />
<transacted/>
<to uri="bean:batchListener?method=init" />
<to uri="seda:queue.process" />
</route>
<route>
<from uri="seda:queue.process" />
<to uri="bean:batchProcessor?method=process" />
</route>
So I came up with this. It fulfills the ability to consume only as many as
needed to process, but I don't think its handling the transaction boundary
well because of the "blockWhenFull" attribute. It appears that the blocking
on the first route is holding the transaction open until it is able to
commit the message to the seda queue.
<route>
<from uri="q:queue:queue/batch/ready" />
<transacted/>
<to uri="bean:batchListener?method=init" />
<to
uri="seda:queue.process?size=2&blockWhenFull=true&concurrentConsumers=1"
/>
</route>
<route>
<from
uri="seda:queue.process?size=2&blockWhenFull=true&concurrentConsumers=1"
/>
<to uri="bean:batchProcessor?method=process" />
</route>
I understand that the seda component is asynchronous, and that we are
essentially using it to clip off the transaction at the route boundary. But
that has its obvious downside of draining the actvemq queue in rapid fashion
and queuing them in seda. It is almost as if I need to us a vm component
instead of seda, but vm will participate in the transaction, which is the
original problem. I have looked into using a throttle on the first route,
but the data size being transferred is very inconsistent, and I cant rely on
the throttle to feed the second route appropriately.
I appreciate your help. If you could please let me know how I might be able
to use camel to achieve what I have mentioned above.
--
View this message in context: http://camel.465427.n5.nabble.com/transactions-that-span-multiple-routes-tp5723735p5725194.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: transactions that span multiple routes
Posted by Willem jiang <wi...@gmail.com>.
Hi,
You can consider to use the seda endpoint instead of direct endpoint in your consumeReadyQueue route.
Then the transaction should be committed when the exchange is routed to seda endpoint.
--
Willem Jiang
Red Hat, Inc.
FuseSource is now part of Red Hat
Web: http://www.fusesource.com | http://www.redhat.com
Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/) (English)
http://jnn.iteye.com (http://jnn.javaeye.com/) (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On Friday, December 7, 2012 at 5:55 AM, erstanl wrote:
> Here is the use case.
>
> 1) Consume message from ActiveMQ queue
> 2) Update database with data derived from step 1
> 3) Download rather large data from database
> 4) Process data from step 3
> 5) Update database with results from step 4
>
> These routes represent that case (I think):
>
> Steps 1, 2
> <route id="consumeReadyQueue">
> <from uri="q:queue:queue/batch/ready" />
> <transacted/>
> <to uri="bean:batchListener?method=init" />
> <to uri="direct:processBatch" />
> </route>
>
> Steps 3, 4, 5
> <route id="processBatch">
> <from uri="direct:processBatch" />
> <to uri="bean:batchProcessor?method=process" />
> </route>
>
> The problem is, I only want steps 1 and 2 (route: consumeReadyQueue) to
> participate in the XA transaction. From what I have gleaned,
> consumeReadyQueue forces processBatch to participate in the transaction.
> This is not good for my case as processBatch can take up to 30 seconds. Is
> there a way to force the commit to happen before processBatch is running?
>
> Maybe I am approaching this wrong. What are my possible options?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/transactions-that-span-multiple-routes-tp5723735.html
> Sent from the Camel - Users mailing list archive at Nabble.com (http://Nabble.com).