Batch insertion issue when shutting down apache camel

[Priyanka <si...@gmail.com>]

I am doing batch insertion in mysql using mybatis.I keep adding messages to
an ArrayList and fire the query when ArrayList size has reached the batch
size.

<route  id="billingRoute">
<from uri="activemq:queue:{{billing.queue}}" />
<log message="Message received from billing queue is ${body}"></log>
<unmarshal ref="gsonBilling"></unmarshal>
<bean beanType="com.bng.upload.processors.GetCdr"
method="process(com.bng.upload.beans.BillingEvent,${exchange})" />
log message="Multicasting  data ${body} to file system and database" />
<multicast>
<pipeline>
<transform>
<method ref="insertionBean" method="billingBatchInsertion"></method>
</transform>
<choice>
<when>
<simple> ${body.size()} == ${properties:batch.size}</simple>
<to uri="mybatis:batchInsertBilling?statementType=InsertList"></to>
<log message="Inserted in billing table :
${in.header.CamelMyBatisResult}"></log>
</when>
</choice>
</pipeline>
<pipeline>
<choice>
<when>
<simple>${properties:billing.write.file} == true</simple>
<setHeader headerName="path">
<simple>${properties:billing.cdr.folder}</simple>
</setHeader>
<log message="Going to write to file : ${body}"></log>
<bean beanType="com.bng.upload.processors.ToFile"
method="process(${exchange},com.bng.upload.beans.BillingCdr)" />
<to uri="file://?fileExist=Append"></to>
</when>
</choice>
</pipeline>
</multicast>
</route>
</routeContext>

We are not using transactions as there are  multiple components included in
route -Files,ActiveMq and Database. The issue is that when the tomcat is
restarted,if the ArrayList size has not reached the batch size,those
messages get lost.Is there any solution to it ?



--
View this message in context: http://camel.465427.n5.nabble.com/Batch-insertion-issue-when-shutting-down-apache-camel-tp5790268.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Batch insertion issue when shutting down apache camel

[Tomohisa Igarashi <tm...@gmail.com>]

Hi,

I would say transaction is the answer for the case like this, you can use transaction synchronization for non-transacted resources.

But... it's worth a try setting CLIENT_ACKNOWLEDGE and invoke acknowledge() after you finish processing. You can access original JMS message by ((JmsMessage)camelMessage).getJmsMessage().

Thanks,
Tomo

On 11/21/2016 08:32 PM, Priyanka wrote:
> I am doing batch insertion in mysql using mybatis.I keep adding messages to
> an ArrayList and fire the query when ArrayList size has reached the batch
> size.
>
> <route  id="billingRoute">
> <from uri="activemq:queue:{{billing.queue}}" />
> <log message="Message received from billing queue is ${body}"></log>
> <unmarshal ref="gsonBilling"></unmarshal>
> <bean beanType="com.bng.upload.processors.GetCdr"
> method="process(com.bng.upload.beans.BillingEvent,${exchange})" />
> log message="Multicasting  data ${body} to file system and database" />
> <multicast>
> <pipeline>
> <transform>
> <method ref="insertionBean" method="billingBatchInsertion"></method>
> </transform>
> <choice>
> <when>
> <simple> ${body.size()} == ${properties:batch.size}</simple>
> <to uri="mybatis:batchInsertBilling?statementType=InsertList"></to>
> <log message="Inserted in billing table :
> ${in.header.CamelMyBatisResult}"></log>
> </when>
> </choice>
> </pipeline>
> <pipeline>
> <choice>
> <when>
> <simple>${properties:billing.write.file} == true</simple>
> <setHeader headerName="path">
> <simple>${properties:billing.cdr.folder}</simple>
> </setHeader>
> <log message="Going to write to file : ${body}"></log>
> <bean beanType="com.bng.upload.processors.ToFile"
> method="process(${exchange},com.bng.upload.beans.BillingCdr)" />
> <to uri="file://?fileExist=Append"></to>
> </when>
> </choice>
> </pipeline>
> </multicast>
> </route>
> </routeContext>
>
> We are not using transactions as there are  multiple components included in
> route -Files,ActiveMq and Database. The issue is that when the tomcat is
> restarted,if the ArrayList size has not reached the batch size,those
> messages get lost.Is there any solution to it ?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Batch-insertion-issue-when-shutting-down-apache-camel-tp5790268.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>