You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Olivier Mallassi <ol...@gmail.com> on 2015/12/30 10:22:06 UTC
[C++/Java Broker] [Transactions]
Hi all
again, my apologies for all these questions.
I am trying to clarify my understanding around transactions support within
qpid and here is my current understanding (based on documentation & JIRA).
I have to say I did not test anything yet.
My use cases all go into the same pattern : (1) writes into a DB &
publishes messages or (2) consumes messages and updates a DB, all of this
being "atomic" (e.g. if the DB writings fail, messages are not sent
etc...), with Java & C++ clients and I am figuring out if I can rely on XA
/ DTX or if I need to find another implementation.
Here is my current understanding, any comments are welcomed.
AMQP 0.10AMQP 1.0Deployment typeC++ API JMS APIC++ API JMS APIHA Java
brokeractive/Passive
cluster of Java brokerslocal transactionokokokokdistributed / XA
transactionsokokokokFederated + HA C++ Brokers- Federated Exchanges +
clustering of CPP Brokers
- persistence (no message loss)local transactionok (
https://issues.apache.org/jira/browse/QPID-3384)
but looks redundant with https://issues.apache.org/jira/browse/QPID-4328
*not sure the backup are asynchronously updated*?ok (
https://issues.apache.org/jira/browse/QPID-4710)?distributed / XA
transactionsko (based on
https://issues.apache.org/jira/browse/QPID-4328)???Dispatch
Router- local tx will be supported (
https://issues.apache.org/jira/browse/DISPATCH-195)
- distributed tx may be supported
So as you can see, I am not clear about distributed transactions support
(AMQP 0.10 or 1.0). Can you help ?
Cheers.
oliv/
Re: [C++/Java Broker] [Transactions]
Posted by aconway <ac...@redhat.com>.
On Wed, 2016-01-06 at 08:51 +0100, Olivier Mallassi wrote:
> Hi all,
>
>
> sorry.
>
> I have tried to summarize all this content in this table
> https://docs.google.com/spreadsheets/d/1JilNwbBSftHxPZymJoaSeSA1uxpp9
> ZwXZyzDM9q6LQA/edit?usp=sharing
>
> Hope that helps
>
> oliv/
>
> PS: thx for the precisions on the link between on
> session/transaction/process.
As Gordon pointed out my initial comments about dispatch and
transactions were over-simplistic.
Dispatch does "link routing" and "message routing"
To support all AMQP 1.0 transaction styles we need to extend "link
routing" to "session routing" so a client can collect together all the
links in a transaction (including the AMQP transaction factory link)
and ensure they are all routed to the same domain (broker). Then
supporting TX or DTX is simple, we just need to forward the transaction
ID info.
Dispatch "message routing" deliberately decouples things so that even
on a single link, messages may be routed to/from many independent
brokers or non-broker endpoints. Simply forwarding AMQP transaction
information doesn't make sense there.
HOWEVER for DTX there might be another option. Dispatch could use AMQP
only to get the XA transaction ID from the local client. We could
forward the XA info without any session context and use it in a new
session context at the remote end. This would let dispatch participate
in arbitrary XA transactions that are not scoped to any single AMQP
broker (even ones that include non AMQP resources.) The trick would be
where to get the XA transaction from in the first place.
It is Friday so this may all be madness.
>
>
> On Tue, Jan 5, 2016 at 12:54 PM, Gordon Sim <gs...@redhat.com> wrote:
>
> > On 01/05/2016 11:42 AM, Robbie Gemmell wrote:
> >
> > > On 4 January 2016 at 19:59, Gordon Sim <gs...@redhat.com> wrote:
> > >
> > > > On 12/30/2015 06:37 PM, aconway wrote:
> > > >
> > > > > I believe dispatch *could* support at least some of the AMQP
> > > > > 1.0 TX and
> > > > > DTX modes even with message routing, we just haven't gotten
> > > > > there yet.
> > > > >
> > > >
> > > >
> > > > For local transactions, any transfer or disposition related to
> > > > the
> > > > transaction needs to be routed to the same process, over the
> > > > same
> > > > connection[1], as that to which the transaction controller link
> > > > is
> > > > routed.
> > > >
> > > > [1] If the multi-ssns-per-txn capability is not supported by
> > > > the process
> > > > in
> > > > question, the same session would have to be used. If the
> > > > multi-txns-per-ssn
> > > > capability was not supported then you would need a unique
> > > > session for
> > > > every
> > > > distinct transaction.
> > > >
> > > >
> > > To clarify that last bit, I asume you mean every currently active
> > > transaction?
> > >
> >
> > Yes, once the transaction is completed the session would be free to
> > be
> > associated with another one.
> >
> >
> >
> > -----------------------------------------------------------------
> > ----
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [C++/Java Broker] [Transactions]
Posted by Gordon Sim <gs...@redhat.com>.
On 01/06/2016 07:51 AM, Olivier Mallassi wrote:
> I have tried to summarize all this content in this table
> https://docs.google.com/spreadsheets/d/1JilNwbBSftHxPZymJoaSeSA1uxpp9ZwXZyzDM9q6LQA/edit?usp=sharing
Some further clarifications:
The c++ qpid::messaging API does not support distributed transactions at
all. Such support would need to be added in conjunction with some XA
transaction manager.
The JMS API defines optional extended interfaces for XA support. These
are implemented only for AMQP 0-10 and are not intended for direct
invocation by application code but rather as part of the integration
into a JEE application server which would be used for running the
application code. This JCA integration work for Qpid has not been widely
used.
The only other way to drive distributed transactions is through the low
level AMQP 0-10 APIs in python or c++ which are now considered
deprecated for application use (these are what is used for the tests
associated with the c++ brokers dtx support).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [C++/Java Broker] [Transactions]
Posted by Olivier Mallassi <ol...@gmail.com>.
Hi all,
sorry.
I have tried to summarize all this content in this table
https://docs.google.com/spreadsheets/d/1JilNwbBSftHxPZymJoaSeSA1uxpp9ZwXZyzDM9q6LQA/edit?usp=sharing
Hope that helps
oliv/
PS: thx for the precisions on the link between on
session/transaction/process.
On Tue, Jan 5, 2016 at 12:54 PM, Gordon Sim <gs...@redhat.com> wrote:
> On 01/05/2016 11:42 AM, Robbie Gemmell wrote:
>
>> On 4 January 2016 at 19:59, Gordon Sim <gs...@redhat.com> wrote:
>>
>>> On 12/30/2015 06:37 PM, aconway wrote:
>>>
>>>> I believe dispatch *could* support at least some of the AMQP 1.0 TX and
>>>> DTX modes even with message routing, we just haven't gotten there yet.
>>>>
>>>
>>>
>>> For local transactions, any transfer or disposition related to the
>>> transaction needs to be routed to the same process, over the same
>>> connection[1], as that to which the transaction controller link is
>>> routed.
>>>
>>> [1] If the multi-ssns-per-txn capability is not supported by the process
>>> in
>>> question, the same session would have to be used. If the
>>> multi-txns-per-ssn
>>> capability was not supported then you would need a unique session for
>>> every
>>> distinct transaction.
>>>
>>>
>> To clarify that last bit, I asume you mean every currently active
>> transaction?
>>
>
> Yes, once the transaction is completed the session would be free to be
> associated with another one.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: [C++/Java Broker] [Transactions]
Posted by Gordon Sim <gs...@redhat.com>.
On 01/05/2016 11:42 AM, Robbie Gemmell wrote:
> On 4 January 2016 at 19:59, Gordon Sim <gs...@redhat.com> wrote:
>> On 12/30/2015 06:37 PM, aconway wrote:
>>> I believe dispatch *could* support at least some of the AMQP 1.0 TX and
>>> DTX modes even with message routing, we just haven't gotten there yet.
>>
>>
>> For local transactions, any transfer or disposition related to the
>> transaction needs to be routed to the same process, over the same
>> connection[1], as that to which the transaction controller link is routed.
>>
>> [1] If the multi-ssns-per-txn capability is not supported by the process in
>> question, the same session would have to be used. If the multi-txns-per-ssn
>> capability was not supported then you would need a unique session for every
>> distinct transaction.
>>
>
> To clarify that last bit, I asume you mean every currently active transaction?
Yes, once the transaction is completed the session would be free to be
associated with another one.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [C++/Java Broker] [Transactions]
Posted by Robbie Gemmell <ro...@gmail.com>.
On 4 January 2016 at 19:59, Gordon Sim <gs...@redhat.com> wrote:
> On 12/30/2015 06:37 PM, aconway wrote:
>>
>> On Wed, 2015-12-30 at 10:22 +0100, Olivier Mallassi wrote:
>>>
>>> Hi all
>>>
>>> again, my apologies for all these questions.
>>>
>>> I am trying to clarify my understanding around transactions support
>>> within
>>> qpid and here is my current understanding (based on documentation &
>>> JIRA).
>>> I have to say I did not test anything yet.
>>
>>
>> Here's what I know:
>>
>> * qpidd C++ without HA: persistence (linearstore), TX, DTX.
>
>
> Though DTX is only supported over AMQP 0-10 at present. As Rob notes, AMQP
> 1.0 does not fully define how distributed transactions are supposed to work.
>
>> * qpidd C++ with HA: persistence, limited TX, no DTX
>> * dispatch link routing: will forward TX/DTX (I think, not tested)
>
>
> I'd be quite surprised if that works, unless there has been specific code
> added to allow it. The transaction context is sent as a special delivery
> state on the transfer or disposition and the way this is exposed in proton
> would require explicit handling to copy the data object associated with a
> delivery. From a quick grep of the dispatch code I don't see any reference
> to the relevant proton method (pn_disposition_data).
>
>> * dispatch message routing: no TX/DTX
>
>
> [...]
>>
>> I believe dispatch *could* support at least some of the AMQP 1.0 TX and
>> DTX modes even with message routing, we just haven't gotten there yet.
>
>
> For local transactions, any transfer or disposition related to the
> transaction needs to be routed to the same process, over the same
> connection[1], as that to which the transaction controller link is routed.
>
> [1] If the multi-ssns-per-txn capability is not supported by the process in
> question, the same session would have to be used. If the multi-txns-per-ssn
> capability was not supported then you would need a unique session for every
> distinct transaction.
>
To clarify that last bit, I asume you mean every currently active transaction?
Robbie
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [C++/Java Broker] [Transactions]
Posted by Gordon Sim <gs...@redhat.com>.
On 12/30/2015 06:37 PM, aconway wrote:
> On Wed, 2015-12-30 at 10:22 +0100, Olivier Mallassi wrote:
>> Hi all
>>
>> again, my apologies for all these questions.
>>
>> I am trying to clarify my understanding around transactions support
>> within
>> qpid and here is my current understanding (based on documentation &
>> JIRA).
>> I have to say I did not test anything yet.
>
> Here's what I know:
>
> * qpidd C++ without HA: persistence (linearstore), TX, DTX.
Though DTX is only supported over AMQP 0-10 at present. As Rob notes,
AMQP 1.0 does not fully define how distributed transactions are supposed
to work.
> * qpidd C++ with HA: persistence, limited TX, no DTX
> * dispatch link routing: will forward TX/DTX (I think, not tested)
I'd be quite surprised if that works, unless there has been specific
code added to allow it. The transaction context is sent as a special
delivery state on the transfer or disposition and the way this is
exposed in proton would require explicit handling to copy the data
object associated with a delivery. From a quick grep of the dispatch
code I don't see any reference to the relevant proton method
(pn_disposition_data).
> * dispatch message routing: no TX/DTX
[...]
> I believe dispatch *could* support at least some of the AMQP 1.0 TX and
> DTX modes even with message routing, we just haven't gotten there yet.
For local transactions, any transfer or disposition related to the
transaction needs to be routed to the same process, over the same
connection[1], as that to which the transaction controller link is routed.
[1] If the multi-ssns-per-txn capability is not supported by the process
in question, the same session would have to be used. If the
multi-txns-per-ssn capability was not supported then you would need a
unique session for every distinct transaction.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [C++/Java Broker] [Transactions]
Posted by aconway <ac...@redhat.com>.
On Wed, 2015-12-30 at 10:22 +0100, Olivier Mallassi wrote:
> Hi all
>
> again, my apologies for all these questions.
>
> I am trying to clarify my understanding around transactions support
> within
> qpid and here is my current understanding (based on documentation &
> JIRA).
> I have to say I did not test anything yet.
Here's what I know:
* qpidd C++ without HA: persistence (linearstore), TX, DTX.
* qpidd C++ with HA: persistence, limited TX, no DTX
* dispatch link routing: will forward TX/DTX (I think, not tested)
* dispatch message routing: no TX/DTX
Notes:
persistence even without TX gives you at-least-once guarantee, if a
message is acknowledged it will not be lost, even if broker crashes and
is recovered.
qpidd + HA plugin allows TX with restrictions, see https://svn.apache.o
rg/viewvc?view=revision&revision=r1701109
No DTX (not impossible but not trivial)
There is another option: you could deploy qpidd *without* the HA plugin
as a cold-standby cluster.
Use a shared persistent store (e.g. GFS) and pacemaker or other cluster
manager
to run exactly one qpidd at a time from the same store.
This would give you hardware redundancy with full TX and DTX.
The down side is recovery time from the persistent store, depends
heavily on how much queue data there is to recover.
I believe dispatch *could* support at least some of the AMQP 1.0 TX and
DTX modes even with message routing, we just haven't gotten there yet.
> My use cases all go into the same pattern : (1) writes into a DB &
> publishes messages or (2) consumes messages and updates a DB, all of
> this
> being "atomic" (e.g. if the DB writings fail, messages are not sent
> etc...), with Java & C++ clients and I am figuring out if I can rely
> on XA
> / DTX or if I need to find another implementation.
>
> Here is my current understanding, any comments are welcomed.
>
> AMQP 0.10AMQP 1.0Deployment typeC++ API JMS APIC++ API JMS APIHA Java
> brokeractive/Passive
> cluster of Java brokerslocal transactionokokokokdistributed / XA
> transactionsokokokokFederated + HA C++ Brokers- Federated Exchanges +
> clustering of CPP Brokers
> - persistence (no message loss)local transactionok (
> https://issues.apache.org/jira/browse/QPID-3384)
>
> but looks redundant with https://issues.apache.org/jira/browse/QPID-4
> 328
>
> *not sure the backup are asynchronously updated*?ok (
> https://issues.apache.org/jira/browse/QPID-4710)?distributed / XA
> transactionsko (based on
> https://issues.apache.org/jira/browse/QPID-4328)???Dispatch
> Router- local tx will be supported (
> https://issues.apache.org/jira/browse/DISPATCH-195)
> - distributed tx may be supported
>
>
> So as you can see, I am not clear about distributed transactions
> support
> (AMQP 0.10 or 1.0). Can you help ?
>
>
> Cheers.
>
> oliv/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: [C++/Java Broker] [Transactions]
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Olivier,
I'm presuming your mail was formatted as a nice table when you sent it...
but the Apache Mailing list seems to have mangled it to a point where, try
as I might, I have no idea what it is saying :-(
Would it be possible to resend in a way that will survive the
transformation into a plain test mail? (you can see how the mail appears
currently by looking here:
http://qpid.2158936.n2.nabble.com/C-Java-Broker-Transactions-td7635717.html)
In terms of the Java Broker (HA cluster and non HA): Distributed
Transactions are supported for AMQP 0-10, local (non distributed)
transactions are supported for all protocols. AMQP 0-9 and AMQP 1.0 do not
define how XA transactions should work. There was work done on how
distributed transactions should be sent using AMQP 1.0, but this was never
finalised. When the protocol work is done it will be easy to add
distributed transaction support for AMQP 1.0 into the Java Broker.
-- Rob
On 30 December 2015 at 09:22, Olivier Mallassi <ol...@gmail.com>
wrote:
> Hi all
>
> again, my apologies for all these questions.
>
> I am trying to clarify my understanding around transactions support within
> qpid and here is my current understanding (based on documentation & JIRA).
> I have to say I did not test anything yet.
>
> My use cases all go into the same pattern : (1) writes into a DB &
> publishes messages or (2) consumes messages and updates a DB, all of this
> being "atomic" (e.g. if the DB writings fail, messages are not sent
> etc...), with Java & C++ clients and I am figuring out if I can rely on XA
> / DTX or if I need to find another implementation.
>
> Here is my current understanding, any comments are welcomed.
>
> AMQP 0.10AMQP 1.0Deployment typeC++ API JMS APIC++ API JMS APIHA Java
> brokeractive/Passive
> cluster of Java brokerslocal transactionokokokokdistributed / XA
> transactionsokokokokFederated + HA C++ Brokers- Federated Exchanges +
> clustering of CPP Brokers
> - persistence (no message loss)local transactionok (
> https://issues.apache.org/jira/browse/QPID-3384)
>
> but looks redundant with https://issues.apache.org/jira/browse/QPID-4328
>
> *not sure the backup are asynchronously updated*?ok (
> https://issues.apache.org/jira/browse/QPID-4710)?distributed / XA
> transactionsko (based on
> https://issues.apache.org/jira/browse/QPID-4328)???Dispatch
> Router- local tx will be supported (
> https://issues.apache.org/jira/browse/DISPATCH-195)
> - distributed tx may be supported
>
>
> So as you can see, I am not clear about distributed transactions support
> (AMQP 0.10 or 1.0). Can you help ?
>
>
> Cheers.
>
> oliv/
>