You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by "Brenneis, Steve" <St...@theclearinghouse.org> on 2021/04/08 16:27:28 UTC
RE: [External] Re: Looking for a transactional solution
Thank you for your answer George.
We are using the XML DSL. When we try something like this:
<route id="route 1">
<from uri="jms:queue:queuein/>
<transacted ref="PROPAGATION_REQUIRES_NEW"/>
<process ref="process1/>
<to uri="direct:nextRoute1/>
</route>
<route id="route 2>
<from uri="direct:nextRoute1"/>
<transacted ref="PROPAGATION_REQUIRED"/>
<process ref="process2"/>
<to uri="direct:nextRoute2"/>
</route>
<route id="route 3">
<from uri="direct:nextRoute2"/>
<transacted ref="PROPAGATION_REQUIRED"/>
<process ref="process3">
<to ref="jms:queue:queueout/>
</route>
If process2 throws an exception, we see that the transaction is rolled back with redelivered = false and the message has disappeared. However, if process1 throws an exception, the transaction is rolled back with redelivered = true and the message is back on the incoming queue.
This is what I meant by a transaction spanning routes. It seems like this approach should work, but we must be using it wrong.
-----Original Message-----
From: George Daswani <ge...@hotmail.com>
Sent: Wednesday, April 7, 2021 4:20 PM
To: users@camel.apache.org
Subject: [External] Re: Looking for a transactional solution
WARNING: This message was received from an EXTERNAL EMAIL ADDRESS.
Examine the URL and use caution BEFORE clicking links or opening any attachments.
Steve,
The transactional EIP supports crossing multiple routes as long as each one of those routes are transactionally aware (transaction
demarcation) and your JMS connection factory is configured properly and
set-up to use a transaction manager. I personally use camel under an OSGI
container (karaf & blueprint)) and transaction demarcation stuff works just fine with the transaction manager that is feature installable with KARAF
(JTA). If you are using something like spring boot, plenty of JTA
transaction managers available (atomikos, bitronix). The JTA transaction manager will allow you to read from one type of message broker (e.q.
ActiveMQ) and write to another type of message broker (e.q. IBM MQ) in one transaction boundary.
https://clicktime.symantec.com/3NL8rtk3CaQa9bJreF5FDn7Vc?u=https%3A%2F%2Fcamel.apache.org%2Fcomponents%2F3.4.x%2Feips%2Ftransactional-client.html
For example.
<bean id="PROPAGATION_REQUIRED"
class="org.apache.camel.spring.spi.SpringTransactionPolicy">
<property name="transactionManager" ref="jmsTransactionManager"/></bean>
<bean id="PROPAGATION_REQUIRES_NEW"
class="org.apache.camel.spring.spi.SpringTransactionPolicy">
<property name="transactionManager" ref="jmsTransactionManager"/>
<property name="propagationBehaviorName"
value="PROPAGATION_REQUIRES_NEW"/></bean>
from("readFromQueue").transacted("PROPAGATION_REQUIRES_NEW").to("direct:doSomeProcessing").to("writeToQ");
from("doSomeProcessing").transacted("PROPAGATION_REQUIRED");
from("writeToQ").transacted(PROPAGATION_REQUIRED");
On Wed, Apr 7, 2021 at 12:28 PM Brenneis, Steve < Steve.Brenneis@theclearinghouse.org> wrote:
> I hope this is the right place to ask this question. If not, maybe
> someone can direct me to the right place.
>
> I have an application that reads messages from a queue, passes them
> off to a content-based router, which then passes them to various
> validation and transformation routes. At the end of each series of
> routes the transformed, validated messages are placed onto another queue.
>
> The problem to be solved is that the messages must be delivered once
> and only once to the outgoing queue. The solution to this would seem
> to be a queue-to-queue XA transaction, but I can't find anything that
> indicates that a transaction could cross route boundaries.
>
> Is there such a solution available? Thanks for any help with this.
>
> Steve Brenneis
>
> This e-mail may contain information that is privileged or
> confidential. If you are not the intended recipient, please delete the
> e-mail and notify us immediately.
>
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and notify us immediately.
Re: [External] Re: Looking for a transactional solution
Posted by Zheng Feng <zf...@redhat.com>.
Hi Brenneis,
Have you ever tried to set "transacted=true" with the incoming queue ? Just
like <from uri="jms:queue:queuein?transacted=true/>
I think this will use a local transaction with JmsTransactionManager and
rollback the message when an exception is thrown.
Hope this is helpful !
On Fri, Apr 9, 2021 at 12:36 AM Brenneis, Steve <
Steve.Brenneis@theclearinghouse.org> wrote:
> Thank you for your answer George.
>
> We are using the XML DSL. When we try something like this:
>
> <route id="route 1">
> <from uri="jms:queue:queuein/>
> <transacted ref="PROPAGATION_REQUIRES_NEW"/>
> <process ref="process1/>
> <to uri="direct:nextRoute1/>
> </route>
>
> <route id="route 2>
> <from uri="direct:nextRoute1"/>
> <transacted ref="PROPAGATION_REQUIRED"/>
> <process ref="process2"/>
> <to uri="direct:nextRoute2"/>
> </route>
>
> <route id="route 3">
> <from uri="direct:nextRoute2"/>
> <transacted ref="PROPAGATION_REQUIRED"/>
> <process ref="process3">
> <to ref="jms:queue:queueout/>
> </route>
>
> If process2 throws an exception, we see that the transaction is rolled
> back with redelivered = false and the message has disappeared. However, if
> process1 throws an exception, the transaction is rolled back with
> redelivered = true and the message is back on the incoming queue.
>
> This is what I meant by a transaction spanning routes. It seems like this
> approach should work, but we must be using it wrong.
>
> -----Original Message-----
> From: George Daswani <ge...@hotmail.com>
> Sent: Wednesday, April 7, 2021 4:20 PM
> To: users@camel.apache.org
> Subject: [External] Re: Looking for a transactional solution
>
>
>
> WARNING: This message was received from an EXTERNAL EMAIL ADDRESS.
> Examine the URL and use caution BEFORE clicking links or opening any
> attachments.
> Steve,
>
> The transactional EIP supports crossing multiple routes as long
> as each one of those routes are transactionally aware (transaction
> demarcation) and your JMS connection factory is configured properly and
> set-up to use a transaction manager. I personally use camel under an OSGI
> container (karaf & blueprint)) and transaction demarcation stuff works
> just fine with the transaction manager that is feature installable with
> KARAF
> (JTA). If you are using something like spring boot, plenty of JTA
> transaction managers available (atomikos, bitronix). The JTA transaction
> manager will allow you to read from one type of message broker (e.q.
> ActiveMQ) and write to another type of message broker (e.q. IBM MQ) in one
> transaction boundary.
>
>
> https://clicktime.symantec.com/3NL8rtk3CaQa9bJreF5FDn7Vc?u=https%3A%2F%2Fcamel.apache.org%2Fcomponents%2F3.4.x%2Feips%2Ftransactional-client.html
>
> For example.
>
> <bean id="PROPAGATION_REQUIRED"
> class="org.apache.camel.spring.spi.SpringTransactionPolicy">
> <property name="transactionManager"
> ref="jmsTransactionManager"/></bean>
> <bean id="PROPAGATION_REQUIRES_NEW"
> class="org.apache.camel.spring.spi.SpringTransactionPolicy">
> <property name="transactionManager" ref="jmsTransactionManager"/>
> <property name="propagationBehaviorName"
> value="PROPAGATION_REQUIRES_NEW"/></bean>
>
>
>
> from("readFromQueue").transacted("PROPAGATION_REQUIRES_NEW").to("direct:doSomeProcessing").to("writeToQ");
>
> from("doSomeProcessing").transacted("PROPAGATION_REQUIRED");
>
> from("writeToQ").transacted(PROPAGATION_REQUIRED");
>
>
>
>
>
> On Wed, Apr 7, 2021 at 12:28 PM Brenneis, Steve <
> Steve.Brenneis@theclearinghouse.org> wrote:
>
> > I hope this is the right place to ask this question. If not, maybe
> > someone can direct me to the right place.
> >
> > I have an application that reads messages from a queue, passes them
> > off to a content-based router, which then passes them to various
> > validation and transformation routes. At the end of each series of
> > routes the transformed, validated messages are placed onto another queue.
> >
> > The problem to be solved is that the messages must be delivered once
> > and only once to the outgoing queue. The solution to this would seem
> > to be a queue-to-queue XA transaction, but I can't find anything that
> > indicates that a transaction could cross route boundaries.
> >
> > Is there such a solution available? Thanks for any help with this.
> >
> > Steve Brenneis
> >
> > This e-mail may contain information that is privileged or
> > confidential. If you are not the intended recipient, please delete the
> > e-mail and notify us immediately.
> >
> This e-mail may contain information that is privileged or confidential. If
> you are not the intended recipient, please delete the e-mail and notify us
> immediately.
>
RE: [External] Re: Looking for a transactional solution
Posted by "Brenneis, Steve" <St...@theclearinghouse.org>.
Hi George.
It appears that when we use the embedded ActiveMQ server, i.e. the vm protocol, it will not negotiate a transaction with IBM MQ. The standalone ActiveMQ broker seems to work.
Thank you for your help.
-----Original Message-----
From: George Daswani <ge...@hotmail.com>
Sent: Thursday, April 8, 2021 11:39 PM
To: users@camel.apache.org
Subject: Re: [External] Re: Looking for a transactional solution
WARNING: This message was received from an EXTERNAL EMAIL ADDRESS.
Examine the URL and use caution BEFORE clicking links or opening any attachments.
Hello Steve,
It seems to work for me, let me know if you want to see a sample
maven project for it. Are you sure that your "jms" component is
configured correctly and has a transactionManager set and has transacted = true?
<bean id="jms" class="org.apache.camel.component.jms.JmsComponent">
<property name="connectionFactory" ref="jmsConnectionFactory" /> <property name="transactionManager" ref="transactionManager" /> <property name="transacted" value="true" /> </bean>
The sample project I created has a scheduled route that writes to the INPUT queue the current date and time every 10 seconds. There's a route that reads from the queue wrapped in a transaction (propagation_requires_new), has a processor that displays the message on the log, then calls another route wrapped in a transaction (propagation_required) which writes to an outgoing queue. At the end of the second route, it checks to see if the date/time written ends with "30" and if so, throws an exception (which will roll back the write to the outgoing queue, bubbling up to the reading
queue). The TransactionErrorHandler (as long as there's no redeliveries
set on the camel side) automatically rolled back the transaction and the message broker automatically redelivers it.
Just a caveat though, some message brokers (e.q. I used ActiveMQ with the
"jms" component) defines a default edelivery policy. ActiveMQ for example
only redeliveries up to six times if you don't override it with "jms.redeliveryPolicy.maximumRedeliveries" URL connection property.
On Thu, Apr 8, 2021 at 9:27 AM Brenneis, Steve < Steve.Brenneis@theclearinghouse.org> wrote:
> Thank you for your answer George.
>
> We are using the XML DSL. When we try something like this:
>
> <route id="route 1">
> <from uri="jms:queue:queuein/>
> <transacted ref="PROPAGATION_REQUIRES_NEW"/>
> <process ref="process1/>
> <to uri="direct:nextRoute1/>
> </route>
>
> <route id="route 2>
> <from uri="direct:nextRoute1"/>
> <transacted ref="PROPAGATION_REQUIRED"/>
> <process ref="process2"/>
> <to uri="direct:nextRoute2"/>
> </route>
>
> <route id="route 3">
> <from uri="direct:nextRoute2"/>
> <transacted ref="PROPAGATION_REQUIRED"/>
> <process ref="process3">
> <to ref="jms:queue:queueout/>
> </route>
>
> If process2 throws an exception, we see that the transaction is rolled
> back with redelivered = false and the message has disappeared.
> However, if
> process1 throws an exception, the transaction is rolled back with
> redelivered = true and the message is back on the incoming queue.
>
> This is what I meant by a transaction spanning routes. It seems like
> this approach should work, but we must be using it wrong.
>
> -----Original Message-----
> From: George Daswani <ge...@hotmail.com>
> Sent: Wednesday, April 7, 2021 4:20 PM
> To: users@camel.apache.org
> Subject: [External] Re: Looking for a transactional solution
>
>
>
> WARNING: This message was received from an EXTERNAL EMAIL ADDRESS.
> Examine the URL and use caution BEFORE clicking links or opening any
> attachments.
> Steve,
>
> The transactional EIP supports crossing multiple routes as
> long as each one of those routes are transactionally aware
> (transaction
> demarcation) and your JMS connection factory is configured properly and
> set-up to use a transaction manager. I personally use camel under an OSGI
> container (karaf & blueprint)) and transaction demarcation stuff works
> just fine with the transaction manager that is feature installable
> with KARAF
> (JTA). If you are using something like spring boot, plenty of JTA
> transaction managers available (atomikos, bitronix). The JTA
> transaction manager will allow you to read from one type of message broker (e.q.
> ActiveMQ) and write to another type of message broker (e.q. IBM MQ) in
> one transaction boundary.
>
>
> https://clicktime.symantec.com/3NL8rtk3CaQa9bJreF5FDn7Vc?u=https%3A%2F
> %2Fcamel.apache.org%2Fcomponents%2F3.4.x%2Feips%2Ftransactional-client
> .html
>
> For example.
>
> <bean id="PROPAGATION_REQUIRED"
> class="org.apache.camel.spring.spi.SpringTransactionPolicy">
> <property name="transactionManager"
> ref="jmsTransactionManager"/></bean>
> <bean id="PROPAGATION_REQUIRES_NEW"
> class="org.apache.camel.spring.spi.SpringTransactionPolicy">
> <property name="transactionManager" ref="jmsTransactionManager"/>
> <property name="propagationBehaviorName"
> value="PROPAGATION_REQUIRES_NEW"/></bean>
>
>
>
> from("readFromQueue").transacted("PROPAGATION_REQUIRES_NEW").to("direc
> t:doSomeProcessing").to("writeToQ");
>
> from("doSomeProcessing").transacted("PROPAGATION_REQUIRED");
>
> from("writeToQ").transacted(PROPAGATION_REQUIRED");
>
>
>
>
>
> On Wed, Apr 7, 2021 at 12:28 PM Brenneis, Steve <
> Steve.Brenneis@theclearinghouse.org> wrote:
>
> > I hope this is the right place to ask this question. If not, maybe
> > someone can direct me to the right place.
> >
> > I have an application that reads messages from a queue, passes them
> > off to a content-based router, which then passes them to various
> > validation and transformation routes. At the end of each series of
> > routes the transformed, validated messages are placed onto another queue.
> >
> > The problem to be solved is that the messages must be delivered once
> > and only once to the outgoing queue. The solution to this would seem
> > to be a queue-to-queue XA transaction, but I can't find anything
> > that indicates that a transaction could cross route boundaries.
> >
> > Is there such a solution available? Thanks for any help with this.
> >
> > Steve Brenneis
> >
> > This e-mail may contain information that is privileged or
> > confidential. If you are not the intended recipient, please delete
> > the e-mail and notify us immediately.
> >
> This e-mail may contain information that is privileged or
> confidential. If you are not the intended recipient, please delete the
> e-mail and notify us immediately.
>
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and notify us immediately.
Re: [External] Re: Looking for a transactional solution
Posted by George Daswani <ge...@hotmail.com>.
Hello Steve,
It seems to work for me, let me know if you want to see a sample
maven project for it. Are you sure that your "jms" component is
configured correctly and has a transactionManager set and has transacted =
true?
<bean id="jms" class="org.apache.camel.component.jms.JmsComponent">
<property name="connectionFactory" ref="jmsConnectionFactory" />
<property name="transactionManager" ref="transactionManager" />
<property name="transacted" value="true" />
</bean>
The sample project I created has a scheduled route that writes to the INPUT
queue the current date and time every 10 seconds. There's a route that
reads from the queue wrapped in a transaction (propagation_requires_new),
has a processor that displays the message on the log, then calls another
route wrapped in a transaction (propagation_required) which writes to an
outgoing queue. At the end of the second route, it checks to see if the
date/time written ends with "30" and if so, throws an exception (which will
roll back the write to the outgoing queue, bubbling up to the reading
queue). The TransactionErrorHandler (as long as there's no redeliveries
set on the camel side) automatically rolled back the transaction and the
message broker automatically redelivers it.
Just a caveat though, some message brokers (e.q. I used ActiveMQ with the
"jms" component) defines a default edelivery policy. ActiveMQ for example
only redeliveries up to six times if you don't override it with
"jms.redeliveryPolicy.maximumRedeliveries" URL connection property.
On Thu, Apr 8, 2021 at 9:27 AM Brenneis, Steve <
Steve.Brenneis@theclearinghouse.org> wrote:
> Thank you for your answer George.
>
> We are using the XML DSL. When we try something like this:
>
> <route id="route 1">
> <from uri="jms:queue:queuein/>
> <transacted ref="PROPAGATION_REQUIRES_NEW"/>
> <process ref="process1/>
> <to uri="direct:nextRoute1/>
> </route>
>
> <route id="route 2>
> <from uri="direct:nextRoute1"/>
> <transacted ref="PROPAGATION_REQUIRED"/>
> <process ref="process2"/>
> <to uri="direct:nextRoute2"/>
> </route>
>
> <route id="route 3">
> <from uri="direct:nextRoute2"/>
> <transacted ref="PROPAGATION_REQUIRED"/>
> <process ref="process3">
> <to ref="jms:queue:queueout/>
> </route>
>
> If process2 throws an exception, we see that the transaction is rolled
> back with redelivered = false and the message has disappeared. However, if
> process1 throws an exception, the transaction is rolled back with
> redelivered = true and the message is back on the incoming queue.
>
> This is what I meant by a transaction spanning routes. It seems like this
> approach should work, but we must be using it wrong.
>
> -----Original Message-----
> From: George Daswani <ge...@hotmail.com>
> Sent: Wednesday, April 7, 2021 4:20 PM
> To: users@camel.apache.org
> Subject: [External] Re: Looking for a transactional solution
>
>
>
> WARNING: This message was received from an EXTERNAL EMAIL ADDRESS.
> Examine the URL and use caution BEFORE clicking links or opening any
> attachments.
> Steve,
>
> The transactional EIP supports crossing multiple routes as long
> as each one of those routes are transactionally aware (transaction
> demarcation) and your JMS connection factory is configured properly and
> set-up to use a transaction manager. I personally use camel under an OSGI
> container (karaf & blueprint)) and transaction demarcation stuff works
> just fine with the transaction manager that is feature installable with
> KARAF
> (JTA). If you are using something like spring boot, plenty of JTA
> transaction managers available (atomikos, bitronix). The JTA transaction
> manager will allow you to read from one type of message broker (e.q.
> ActiveMQ) and write to another type of message broker (e.q. IBM MQ) in one
> transaction boundary.
>
>
> https://clicktime.symantec.com/3NL8rtk3CaQa9bJreF5FDn7Vc?u=https%3A%2F%2Fcamel.apache.org%2Fcomponents%2F3.4.x%2Feips%2Ftransactional-client.html
>
> For example.
>
> <bean id="PROPAGATION_REQUIRED"
> class="org.apache.camel.spring.spi.SpringTransactionPolicy">
> <property name="transactionManager"
> ref="jmsTransactionManager"/></bean>
> <bean id="PROPAGATION_REQUIRES_NEW"
> class="org.apache.camel.spring.spi.SpringTransactionPolicy">
> <property name="transactionManager" ref="jmsTransactionManager"/>
> <property name="propagationBehaviorName"
> value="PROPAGATION_REQUIRES_NEW"/></bean>
>
>
>
> from("readFromQueue").transacted("PROPAGATION_REQUIRES_NEW").to("direct:doSomeProcessing").to("writeToQ");
>
> from("doSomeProcessing").transacted("PROPAGATION_REQUIRED");
>
> from("writeToQ").transacted(PROPAGATION_REQUIRED");
>
>
>
>
>
> On Wed, Apr 7, 2021 at 12:28 PM Brenneis, Steve <
> Steve.Brenneis@theclearinghouse.org> wrote:
>
> > I hope this is the right place to ask this question. If not, maybe
> > someone can direct me to the right place.
> >
> > I have an application that reads messages from a queue, passes them
> > off to a content-based router, which then passes them to various
> > validation and transformation routes. At the end of each series of
> > routes the transformed, validated messages are placed onto another queue.
> >
> > The problem to be solved is that the messages must be delivered once
> > and only once to the outgoing queue. The solution to this would seem
> > to be a queue-to-queue XA transaction, but I can't find anything that
> > indicates that a transaction could cross route boundaries.
> >
> > Is there such a solution available? Thanks for any help with this.
> >
> > Steve Brenneis
> >
> > This e-mail may contain information that is privileged or
> > confidential. If you are not the intended recipient, please delete the
> > e-mail and notify us immediately.
> >
> This e-mail may contain information that is privileged or confidential. If
> you are not the intended recipient, please delete the e-mail and notify us
> immediately.
>