Need for XA?

[Tom Fornoville <to...@roots.be>]

Hi camel-riders,

Happy New Year!

In our application we have a lot of routes that follow the same pattern:
read a message, do some processing, update/insert in the database and then
send a message to the next route.

You can see an example here:

<route id="div-validate-line">
<from uri="activemq:div.line"/>
<bean ref="divServiceActivator" method="validateDivLine"/>
<choice>
<when>
<simple>${header.lineValidationResponse} == 'VALID'</simple>
<bean ref="divServiceActivator" method="persistDivVehicle"/>
</when>
</choice>
<to uri="activemq:div.line.processed"/>
</route>

Coming from the web-applications world we all thought that we would need XA
transactions to make sure we never lose messages (or lines in the example
above).

Working with Camel the last few months (JBoss Fuse 6.0 in particular) we
found that it's not easy setting up a transaction manager : Aries, bundled
with JBoss Fuse, doesn't support pooled XA resources and we can't seem to
get Atomikos working in Fuse.

So we took a step back and asked whether we even need XA transactions. Couldn't
we just define an errorhandler that requeues the message and after x
attempts sends it to a dead letter queue? That way the transactionmanager
will rollback everything that was done in the database (in the
persisDivVehicle method in the above example) and we don't lose any lines.

So the question is: do we need XA transactions or can we ensure that we
don't lose messages by some usage op errorhandler and/or properties on the
route or endpoints?

Thanks in advance for your feedback.

Cheers,
Tom

Re: Need for XA?

[Henryk Konsek <he...@gmail.com>]

> I meant XA processing

Switching from local transactions to XA in Camel is not really
complicated. All you have to do is switching your connection factories
and transaction manager to XA-aware ones. And of course making sure
that you don't have certain options enabled on Camel endpoints (like
JMS consumer caching or transaction timeout).

Here [1] is the documentation from Fuse team related to XA. Here [2]
is the example from Torsten Mielke (fellow of mine from Red Hat). I
think that Christian Mueller also covers XA in his examples [3] from
the Apache Conf. At last but not least, there is a section related to
XA in Camel Bible [4].

I'm also working on the Pax Exam based XA demo making use of Aries JTA
[5] support. I'll publish it on the GitHub when it is done.

Cheers.

[1] https://access.redhat.com/site/documentation/en-US/Fuse_ESB_Enterprise/7.1/html/EIP_Transaction_Guide/files/XaJms-Overview.html
[2] https://github.com/tmielke/fuse-demos/tree/master/Camel/Camel-JMS-JDBC-XA-TX
[3] https://github.com/muellerc/camel-in-transaction
[4] http://www.amazon.co.uk/Camel-Action-Claus-Ibsen/dp/1935182366/ref=sr_1_1?ie=UTF8&qid=1389097895&sr=8-1&keywords=camel+in+action
[5] http://aries.apache.org/modules/transactionsproject.html

-- 
Henryk Konsek
http://henryk-konsek.blogspot.com

Re: Need for XA?

[Krzysztof Sobkowiak <kr...@gmail.com>]

Hi

I meant XA processing

Krzysztof

On 06.01.2014 22:32, Henryk Konsek wrote:
> Krzysztof said:
>> Are there any samples how to develop such requirement with camel?
> You mean XA or non-XA processing? Maybe I could find something for you :) .
>


-- 
Krzysztof Sobkowiak

JEE & OSS Architect | Technical Architect @ Capgemini
e-mail: krzys.sobkowiak@gmail.com <ma...@gmail.com> | 
Twitter: @KSobkowiak

Re: Need for XA?

[Henryk Konsek <he...@gmail.com>]

Krzysztof said:
> Are there any samples how to develop such requirement with camel?

You mean XA or non-XA processing? Maybe I could find something for you :) .

-- 
Henryk Konsek
http://henryk-konsek.blogspot.com

Re: Need for XA?

[Krzysztof Sobkowiak <kr...@gmail.com>]

Hi

Which page do you mean?

Krzysztof

On 05.01.2014 07:28, kraythe . wrote:
> the camel page on transactions shows you how to do an XA transaction
> manager. :)
>
> *Robert Simmons Jr. MSc. - Lead Java Architect @ EA*
> *Author of: Hardcore Java (2003) and Maintainable Java (2012)*
> *LinkedIn: **http://www.linkedin.com/pub/robert-simmons/40/852/a39
> <http://www.linkedin.com/pub/robert-simmons/40/852/a39>*

Re: Need for XA?

["kraythe ." <kr...@gmail.com>]

the camel page on transactions shows you how to do an XA transaction
manager. :)

*Robert Simmons Jr. MSc. - Lead Java Architect @ EA*
*Author of: Hardcore Java (2003) and Maintainable Java (2012)*
*LinkedIn: **http://www.linkedin.com/pub/robert-simmons/40/852/a39
<http://www.linkedin.com/pub/robert-simmons/40/852/a39>*


On Sat, Jan 4, 2014 at 5:10 PM, Krzysztof Sobkowiak <
krzys.sobkowiak@gmail.com> wrote:

> Hi
>
> Are there any samples how to develop such requirement with camel?
>
> Krzysztof
>
>
> On 04.01.2014 22:04, Pontus Ullgren wrote:
>
>> It all depends on your requirements and how your database and the
>> system consuming from the second queue is designed.
>>
>> If the write operation to the database and the system consuming from
>> the second queue is idempotent, ie they can both gracefully handle
>> that the same message is processed twice. Then you can think about
>> skipping XA.
>>
>> If this is NOT the case, or if you are unsure what would happen if the
>> same message is consumed twice, then you will need XA.
>>
>> In the scenario you describe I would defiantly consider using XA just
>> to be safe.
>>
>> Pontus
>>
>> On Fri, Jan 3, 2014 at 11:48 PM, Larry Meadors <la...@gmail.com>
>> wrote:
>>
>>> Just to be clear here...if you're doing this:
>>>   - get a message from JMS broker A
>>>   - write to database
>>>   - put a messages in a queue on JMS broker A
>>>
>>> Should you or should you not need XA?
>>> Larry
>>>
>>
>
> --
> Krzysztof Sobkowiak
>
> JEE & OSS Architect | Technical Architect @ Capgemini
> e-mail: krzys.sobkowiak@gmail.com <ma...@gmail.com> |
> Twitter: @KSobkowiak
>

Re: Need for XA?

[Krzysztof Sobkowiak <kr...@gmail.com>]

Hi

Are there any samples how to develop such requirement with camel?

Krzysztof

On 04.01.2014 22:04, Pontus Ullgren wrote:
> It all depends on your requirements and how your database and the
> system consuming from the second queue is designed.
>
> If the write operation to the database and the system consuming from
> the second queue is idempotent, ie they can both gracefully handle
> that the same message is processed twice. Then you can think about
> skipping XA.
>
> If this is NOT the case, or if you are unsure what would happen if the
> same message is consumed twice, then you will need XA.
>
> In the scenario you describe I would defiantly consider using XA just
> to be safe.
>
> Pontus
>
> On Fri, Jan 3, 2014 at 11:48 PM, Larry Meadors <la...@gmail.com> wrote:
>> Just to be clear here...if you're doing this:
>>   - get a message from JMS broker A
>>   - write to database
>>   - put a messages in a queue on JMS broker A
>>
>> Should you or should you not need XA?
>> Larry


-- 
Krzysztof Sobkowiak

JEE & OSS Architect | Technical Architect @ Capgemini
e-mail: krzys.sobkowiak@gmail.com <ma...@gmail.com> | 
Twitter: @KSobkowiak

Re: Need for XA?

[Henryk Konsek <he...@gmail.com>]

> If the write operation to the database and the system consuming from
> the second queue is idempotent, ie they can both gracefully handle
> that the same message is processed twice. Then you can think about
> skipping XA.

100% true.

Keep also in mind that even if the DB service itself is not
idempotent, you can usually still make it such be using Camel's
idempotent consumer [1] pattern.

> In the scenario you describe I would defiantly consider using XA just
> to be safe.

Actually I can't agree with the sentence above. If the resources
taking part in the transaction are idempotent (DB service and second
queue in discusses case) then you can safely resign from XA. There is
no place for "just to be safe" here, as idempotent-based processing
model is 100% safe.

Moreover, I would even say that you *should* resign from XA whenever
it is possible. XA transactions come with really heavy performance,
latency and scalability penalty. In case of Camel it is not only the
matter of the raw XA performance issues, but also JMS component [2]
performance (Camel can't cache JMS consumers while using XA). Actually
even the one of the most famous articles dedicated for messaging
(Starbucks Does Not Use Two-Phase Commit [3] by G. Hohpe) basically
says that XA should be avoided on the favor of the idempotent
consumers and local transactions.

In my humble opinion you should have really good reason to use heavy
XA, instead of idempotent-based processing which perform and scales
like a hell. Creating bottlenecks in the system should have really
good explanation :) .

Cheers.

[1] http://camel.apache.org/idempotent-consumer.html
[2] http://camel.apache.org/jms.html
[3] http://www.eaipatterns.com/ramblings/18_starbucks.html

-- 
Henryk Konsek
http://henryk-konsek.blogspot.com

Re: Need for XA?

[Larry Meadors <la...@gmail.com>]

OK, thanks - that is how our system was configured, so it was right -
I was just having second thoughts. :)

On Sat, Jan 4, 2014 at 2:04 PM, Pontus Ullgren <ul...@gmail.com> wrote:
> It all depends on your requirements and how your database and the
> system consuming from the second queue is designed.
>
> If the write operation to the database and the system consuming from
> the second queue is idempotent, ie they can both gracefully handle
> that the same message is processed twice. Then you can think about
> skipping XA.
>
> If this is NOT the case, or if you are unsure what would happen if the
> same message is consumed twice, then you will need XA.
>
> In the scenario you describe I would defiantly consider using XA just
> to be safe.
>
> Pontus
>
> On Fri, Jan 3, 2014 at 11:48 PM, Larry Meadors <la...@gmail.com> wrote:
>> Just to be clear here...if you're doing this:
>>  - get a message from JMS broker A
>>  - write to database
>>  - put a messages in a queue on JMS broker A
>>
>> Should you or should you not need XA?
>> Larry

Re: Need for XA?

[Pontus Ullgren <ul...@gmail.com>]

It all depends on your requirements and how your database and the
system consuming from the second queue is designed.

If the write operation to the database and the system consuming from
the second queue is idempotent, ie they can both gracefully handle
that the same message is processed twice. Then you can think about
skipping XA.

If this is NOT the case, or if you are unsure what would happen if the
same message is consumed twice, then you will need XA.

In the scenario you describe I would defiantly consider using XA just
to be safe.

Pontus

On Fri, Jan 3, 2014 at 11:48 PM, Larry Meadors <la...@gmail.com> wrote:
> Just to be clear here...if you're doing this:
>  - get a message from JMS broker A
>  - write to database
>  - put a messages in a queue on JMS broker A
>
> Should you or should you not need XA?
> Larry

Re: Need for XA?

[Larry Meadors <la...@gmail.com>]

Just to be clear here...if you're doing this:
 - get a message from JMS broker A
 - write to database
 - put a messages in a queue on JMS broker A

Should you or should you not need XA?
Larry

Re: Need for XA?

["kraythe ." <kr...@gmail.com>]

XA transactions are usually only needed if you have multisystem
transactions to be coordinated. So if you read from a JMS queue and because
of that store a message in a database then send a message on to another
queue, you want both the DB and the JMS to roll back if there is a 500
error calling a web site or something. For that you need XA. If you are
just going broker queue to queue tyou dont need XA.

*Robert Simmons Jr. MSc. - Lead Java Architect @ EA*
*Author of: Hardcore Java (2003) and Maintainable Java (2012)*
*LinkedIn: **http://www.linkedin.com/pub/robert-simmons/40/852/a39
<http://www.linkedin.com/pub/robert-simmons/40/852/a39>*


On Fri, Jan 3, 2014 at 3:33 AM, Henryk Konsek <he...@gmail.com> wrote:

> Hi Tom,
>
> > In our application we have a lot of routes that follow the same pattern:
> > read a message, do some processing, update/insert in the database and
> then
> > send a message to the next route.
> > (...)
> > do we need XA transactions (...) ?
>
> That's right. You don't need XA to transactionally process your
> message flow. Plain local JMS transaction, ActiveMQ reredelivery
> mechanism and some Camel magic will be fine here. :)
>
> Just mark consumer endpoint as transactional [1] (as demonstrated on
> the snippet below) and ActiveMQ will handle the message redelivery for
> you.
>
> <from uri="activemq:div.line"/>
> <transacted/>
> <bean ref="divServiceActivator" method="validateDivLine"/>
>
> Keep in mind that ActiveMQ broker will keep the message in div.line
> queue until the transaction will be committed by the Camel at the end
> of the route (after sending message to div.line.processed queue). If
> divServiceActivator will throw the exception during the DB update, the
> Camel will not commit and message incoming from div.line will be
> redelivered by the broker. You can control how many times message will
> be redelivered by the broker on the ActiveMQ configuration level [2].
>
> Now is the tricky part. Notice that it is still possible that Camel
> can fail to commit JMS transaction *after* successful DB update (power
> outages happen). That would mean that incoming message will be
> redelivered and processed twice. To handle this situation and keep
> your system state concise make divServiceActivator#persistDivVehicle()
> bean invocation idempotent [3], so it won't try to alter database
> twice. You can also try to make
> divServiceActivator#persistDivVehicle() operation idempotent on the
> business logic level (sometimes it is as trivial as checking if entity
> with given ID hasn't been persisted already).
>
> Cheers.
>
> [1] http://camel.apache.org/transactional-client.html
> [2] http://activemq.apache.org/message-redelivery-and-dlq-handling.html
> [3] http://camel.apache.org/idempotent-consumer.html
>
> --
> Henryk Konsek
> http://henryk-konsek.blogspot.com
>

Re: Need for XA?

[Henryk Konsek <he...@gmail.com>]

Hi Tom,

> In our application we have a lot of routes that follow the same pattern:
> read a message, do some processing, update/insert in the database and then
> send a message to the next route.
> (...)
> do we need XA transactions (...) ?

That's right. You don't need XA to transactionally process your
message flow. Plain local JMS transaction, ActiveMQ reredelivery
mechanism and some Camel magic will be fine here. :)

Just mark consumer endpoint as transactional [1] (as demonstrated on
the snippet below) and ActiveMQ will handle the message redelivery for
you.

<from uri="activemq:div.line"/>
<transacted/>
<bean ref="divServiceActivator" method="validateDivLine"/>

Keep in mind that ActiveMQ broker will keep the message in div.line
queue until the transaction will be committed by the Camel at the end
of the route (after sending message to div.line.processed queue). If
divServiceActivator will throw the exception during the DB update, the
Camel will not commit and message incoming from div.line will be
redelivered by the broker. You can control how many times message will
be redelivered by the broker on the ActiveMQ configuration level [2].

Now is the tricky part. Notice that it is still possible that Camel
can fail to commit JMS transaction *after* successful DB update (power
outages happen). That would mean that incoming message will be
redelivered and processed twice. To handle this situation and keep
your system state concise make divServiceActivator#persistDivVehicle()
bean invocation idempotent [3], so it won't try to alter database
twice. You can also try to make
divServiceActivator#persistDivVehicle() operation idempotent on the
business logic level (sometimes it is as trivial as checking if entity
with given ID hasn't been persisted already).

Cheers.

[1] http://camel.apache.org/transactional-client.html
[2] http://activemq.apache.org/message-redelivery-and-dlq-handling.html
[3] http://camel.apache.org/idempotent-consumer.html

-- 
Henryk Konsek
http://henryk-konsek.blogspot.com