transaction handling while moving data from queue to seda to queue

[amitmahesh <la...@yahoo.com>]

We are developing system which processes transactions coming from ATM
machines of various banks.
We take incoming messages to Request Queue, Process it at multiple stages
and Send response back to Response Queue which is then sent back to Socket
connection established when ATM initiated transaction.
I have some questions related to design of this system using ActiveMQ and
Camel.

Q1) Do I need a JTA transaction or regular JMS transaction when I move data
from one queue to another queue of same broker. Assume no Database is
involved and queues are in persistent mode. Since both Queue are associated
with same broker  so both comes under same Resource Manager, so want to
clarify if XA type transaction is involved over here ? 

Q2) Assume I have 3 stages between Request and Response Queue, each stage
might do some data processing like calling a remote service or calling some
local api etc. So do I need to keep a SEDA block for each stage or a regular
pipeline block will suffice. If I consume aysnchronously from Request queue
with some concurrent consumers then I would have already got that many
threads to work on those stages so whats the necessity of introducing SEDA
stages ?? Also SEDA stages might also complicate jms transaction rollback if
any of the stage throws exception. What I want is the data to move from
REquest to Response only when all 3 intermediate stages completes otherwise
message to be redelivered before sending it to DLQ. Only benefit which I see
in using SEDA is that I can declare async concurrent consumers at every
stages, but as I mentioned earlier I already have concurrent consumer
threads consuming from Request Queue and calling these intermediate Services
in pipeline block.

Q3) Assuming I need to save intermediate data in database at every stages of
processing, we dont want to disturb the main loop of request and response
queue bcoz performance is the key in this system, so we thought of sending
intermediate data in a separate queue dedicated for just Inserts/Updates. So
for every transaction I will send a insert message when it reaches the
Request queue and then multiple update messages on completion of each
stages. Failure at Database queue should not stop the main loop. So assuming
incoming transaction message was pipe delimited text, then I convert it to
JPA entity object and send it to DB queue for insert, then after every stage
I update this entity object and send it to DB queue which in turn calls
EntityManager.merge() to update previously inserted record. Will this work
Or we should create entities only after consuming text from DB queue. 










--
View this message in context: http://camel.465427.n5.nabble.com/transaction-handling-while-moving-data-from-queue-to-seda-to-queue-tp5775247.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: transaction handling while moving data from queue to seda to queue

[FingolfinTEK <fi...@gmail.com>]

amitmahesh wrote
> Q1) Do I need a JTA transaction or regular JMS transaction when I move
> data from one queue to another queue of same broker. Assume no Database is
> involved and queues are in persistent mode. Since both Queue are
> associated with same broker  so both comes under same Resource Manager, so
> want to clarify if XA type transaction is involved over here ? 

>From my perspective, a local transaction should suffice if you are using the
same broker, no need to involve XA.


amitmahesh wrote
> Q2) Assume I have 3 stages between Request and Response Queue, each stage
> might do some data processing like calling a remote service or calling
> some local api etc. So do I need to keep a SEDA block for each stage or a
> regular pipeline block will suffice. If I consume aysnchronously from
> Request queue with some concurrent consumers then I would have already got
> that many threads to work on those stages so whats the necessity of
> introducing SEDA stages ?? Also SEDA stages might also complicate jms
> transaction rollback if any of the stage throws exception. What I want is
> the data to move from REquest to Response only when all 3 intermediate
> stages completes otherwise message to be redelivered before sending it to
> DLQ. Only benefit which I see in using SEDA is that I can declare async
> concurrent consumers at every stages, but as I mentioned earlier I already
> have concurrent consumer threads consuming from Request Queue and calling
> these intermediate Services in pipeline block.

If you introduce SEDA endpoints into your route(s), they will not be
included into your current transaction, regardless of whether you use
asynchronous SEDA or not. As far as I know, SEDA transactions are planned
for Camel 3.0. My advice would be to use the  Direct component
<http://camel.apache.org/direct.html>   to link your routes together instead
of SEDA.


amitmahesh wrote
> Q3) Assuming I need to save intermediate data in database at every stages
> of processing, we dont want to disturb the main loop of request and
> response queue bcoz performance is the key in this system, so we thought
> of sending intermediate data in a separate queue dedicated for just
> Inserts/Updates. So for every ATM message I will send a insert message
> when it reaches the Request queue and then multiple update messages on
> completion of each stages. Failure at Database queue should not stop the
> main loop. So assuming incoming message was pipe delimited text, then I
> convert it to JPA entity object and send it to DB queue for insert, then
> after every stage I update this entity object and send it to DB queue
> which in turn calls EntityManager.merge() to update previously inserted
> record. Will this work Or we should create entities only after consuming
> text from DB queue.  Does overall approach looks feasible or we are
> missing anything ??

Sounds feasible although you should probably make sure there are no
race-conditions when consuming from the DB queue - if you have multiple
consumers (either concurrent ones or multiple instances of your application
running on different servers) and send out a series of messages, e.g.
INSERT, UPDATE1, UPDATE2, UPDATE3 it is possible that the order in which you
consume them will not be the same and you end up with INSERT, UPDATE1,
UPDATE3, UPDATE2, resulting in lost updates. My advice would be to benchmark
your performance first and then decide whether it is necessary to move
database operations out of your route since the approach you describe here
introduces additional complexity and considerations.


amitmahesh wrote
> Q4) Currently we support 5 banks so we have 5 Req/Resp queue pairs, in
> future we anticipate adding new banks, we need to design the system to add
> those additional future Queues/Routes without stopping the running routes.
> Is that possible with JMX/CamelContext api or we need some OSGi/Karaf
> interface ??

I'm not sure if it is possible to do via JMX - I don't think so but even if
it is, it's a really, really, REALLY bad idea - the equivalent of manually
changing a database schema of a running production application. Either think
about OSGi/Karaf or just change your deployment procedure and don't stop the
old version of the application until the new one is up and running. 




--
View this message in context: http://camel.465427.n5.nabble.com/transaction-handling-while-moving-data-from-queue-to-seda-to-queue-tp5775247p5775457.html
Sent from the Camel - Users mailing list archive at Nabble.com.