You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Arnaud Simon <as...@redhat.com> on 2007/03/26 13:36:20 UTC

dtx API

Hello,

I have attached with this email a proposal for APIs that I expect to use
for dtx support. Those interfaces are the one I expect to implement on
the Java broker. The C++ broker may use some slightly different APIs but
they should semantically be very similar. 
Those APIs have been designed to: 
1) make a clean distinction between transaction manager, recovery
manager and persistence store. 
2) Use whenever possible a direct mapping between dtx classes and
transaction manager methods. (you'll see that exceptions should be
mapped to channel close with the corresponding error code and results
can be returned straight ahead) 
3) Use an unique way of identifying transactional work by only
propagating Xids all over the place. The underlying implementation can
keep the mapping between its tx context representation and Xids. 

I would very much appreciate your comments. 

For a synchronization purpose I am going to do the following unless it
clashes with somebody else's agenda: 
- on the broker:
1) refactor the current code for supporting the proposed interfaces 
2) add dtx classes support
3) extend current dbd code for dtx support
4) add support for dtx around a file based store
-on the client:
1) add dtx classes support 
2) extend JMS implementation with XA capabilities

I should start with the broker (1, 2 and 3) then with the client and
finally do 4) on the broker. 

Regards

Arnaud 


Re: dtx API

Posted by Arnaud Simon <as...@redhat.com>.
Hi Rob,

I cannot commit code yet. So I would suggest that it could be easier if
I work with you in the sense that I would ask you to commit my patches.
If we do that then we should avoid any conflict or whatsoever. 
When have you planned to do the merge? 
Please let me know what you think about that.

Regards

Arnaud 

On Mon, 2007-03-26 at 12:57 +0100, Robert Godfrey wrote:
> Arnaud,
> 
> Not sure if the attachment got removed at some time, but I can't see
> it...
> 
> Anyway, before you start work on the Java Broker... I'm currently
> doing a major refactoring so that we can run multiple versions in
> parallel - I'm doing this on a separate branch for the moment, but
> would appreciate it if the trunk code didn't change too much while I'm
> at it as it will make the merge harder :-)  After my work is completed
> we'll be moving down the 0-9 support into the Java side, and then
> start working on 0-10 support.  dtx will only be in AMQP 0-10, so
> really we should be aiming to do it with that work... howevere,i f
> you're keen to get cracking straight away, maybe you could start
> another branch, at least until the dust settles with M2 and trunk. 
> 
> -- Rob
> 
> 
> 
> On 26/03/07, Arnaud Simon <as...@redhat.com> wrote:
>         Hello,
>         
>         I have attached with this email a proposal for APIs that I
>         expect to use
>         for dtx support. Those interfaces are the one I expect to
>         implement on
>         the Java broker. The C++ broker may use some slightly
>         different APIs but 
>         they should semantically be very similar.
>         Those APIs have been designed to:
>         1) make a clean distinction between transaction manager,
>         recovery
>         manager and persistence store.
>         2) Use whenever possible a direct mapping between dtx classes
>         and 
>         transaction manager methods. (you'll see that exceptions
>         should be
>         mapped to channel close with the corresponding error code and
>         results
>         can be returned straight ahead)
>         3) Use an unique way of identifying transactional work by
>         only 
>         propagating Xids all over the place. The underlying
>         implementation can
>         keep the mapping between its tx context representation and
>         Xids.
>         
>         I would very much appreciate your comments.
>         
>         For a synchronization purpose I am going to do the following
>         unless it 
>         clashes with somebody else's agenda:
>         - on the broker:
>         1) refactor the current code for supporting the proposed
>         interfaces
>         2) add dtx classes support
>         3) extend current dbd code for dtx support
>         4) add support for dtx around a file based store 
>         -on the client:
>         1) add dtx classes support
>         2) extend JMS implementation with XA capabilities
>         
>         I should start with the broker (1, 2 and 3) then with the
>         client and
>         finally do 4) on the broker.
>         
>         Regards 
>         
>         Arnaud
>         
> 


Re: dtx API

Posted by Robert Godfrey <ro...@gmail.com>.
Arnaud,

Not sure if the attachment got removed at some time, but I can't see it...

Anyway, before you start work on the Java Broker... I'm currently doing a
major refactoring so that we can run multiple versions in parallel - I'm
doing this on a separate branch for the moment, but would appreciate it if
the trunk code didn't change too much while I'm at it as it will make the
merge harder :-)  After my work is completed we'll be moving down the 0-9
support into the Java side, and then start working on 0-10 support.  dtx
will only be in AMQP 0-10, so really we should be aiming to do it with that
work... howevere,i f you're keen to get cracking straight away, maybe you
could start another branch, at least until the dust settles with M2 and
trunk.

-- Rob



On 26/03/07, Arnaud Simon <as...@redhat.com> wrote:
>
> Hello,
>
> I have attached with this email a proposal for APIs that I expect to use
> for dtx support. Those interfaces are the one I expect to implement on
> the Java broker. The C++ broker may use some slightly different APIs but
> they should semantically be very similar.
> Those APIs have been designed to:
> 1) make a clean distinction between transaction manager, recovery
> manager and persistence store.
> 2) Use whenever possible a direct mapping between dtx classes and
> transaction manager methods. (you'll see that exceptions should be
> mapped to channel close with the corresponding error code and results
> can be returned straight ahead)
> 3) Use an unique way of identifying transactional work by only
> propagating Xids all over the place. The underlying implementation can
> keep the mapping between its tx context representation and Xids.
>
> I would very much appreciate your comments.
>
> For a synchronization purpose I am going to do the following unless it
> clashes with somebody else's agenda:
> - on the broker:
> 1) refactor the current code for supporting the proposed interfaces
> 2) add dtx classes support
> 3) extend current dbd code for dtx support
> 4) add support for dtx around a file based store
> -on the client:
> 1) add dtx classes support
> 2) extend JMS implementation with XA capabilities
>
> I should start with the broker (1, 2 and 3) then with the client and
> finally do 4) on the broker.
>
> Regards
>
> Arnaud
>
>

Re: [c++] Re: dtx API

Posted by Arnaud Simon <as...@redhat.com>.
On Mon, 2007-03-26 at 17:46 +0100, Rupert Smith wrote:
> Are you planning on using the interfaces already provided in
> javax.transaction.xa? I noticed that your TransactionManager interface
> is very close to XAResource and your Xid interface is identical to the
> one in that package. Is this stuff working at a lower level or is it
> intended to be a JTA implementation that works in the standard way
> with java transaction managers? 

The plan is to provide XA support i.e. we will implement an XA Resource
on the client side and most certainly a JMS XAConnection and
corresponding XASession. AMPQ defines the dtx classes for providing XA
support this is why on the server, the TM API is similar to the XA one
you are familiar with.

> The reason I ask, is I'm trying to imagine writing a JCA resource
> adapter to plug Qpid into an app server. To do this you have to
> implement XAResource in javax.transaction.xa.

We will need to implement JMS XAConnection, XASession and XAResource and
ConnectionConsumer for plugging our JMS implementation in an App server.
This is the case when we want to use JMS but I don't think we should try
to plug Qpid within an App server using a resource adapter. 

> Also, what about the case for non-XA (local only) transactions. The
> transactional enqueue and dequeue methods must take Xid parameters. In
> the local case would you just generate some suitable values to use for
> these or would the local transaction methods on MessageStore be
> different or just like they currently are? 

We will use the same methods, the only difference would be that the API
caller would need to create an Xid for referencing a local tx. So, there
isn't a real difference between a local tx and a distributed one. The
only thing is that local transactions always use one phase commit. So
you would do something like: 
xid = Xid.create()
TM.begin(xid)
PS.enqueue(xid, M)
TM.commit-one-phase(xid)




Re: [c++] Re: dtx API

Posted by Rupert Smith <ru...@googlemail.com>.
Are you planning on using the interfaces already provided in
javax.transaction.xa? I noticed that your TransactionManager interface is
very close to XAResource and your Xid interface is identical to the one in
that package. Is this stuff working at a lower level or is it intended to be
a JTA implementation that works in the standard way with java transaction
managers?

The reason I ask, is I'm trying to imagine writing a JCA resource adapter to
plug Qpid into an app server. To do this you have to implement XAResource in
javax.transaction.xa.

Also, what about the case for non-XA (local only) transactions. The
transactional enqueue and dequeue methods must take Xid parameters. In the
local case would you just generate some suitable values to use for these or
would the local transaction methods on MessageStore be different or just
like they currently are?

Rupert

On 26/03/07, Arnaud Simon <as...@redhat.com> wrote:
>
> +1
>
> I first though about only sending bytes arrays and Uids around but it
> was too restrictive so we went for what we currently have.
> I´ll synchronize with Gordon for defining PersistableQueue and
> PersistableMessage interfaces.
>
> Cheers
>
> Arnaud
>
>
> On Mon, 2007-03-26 at 14:42 +0100, Gordon Sim wrote:
> > Alan Conway wrote:
> > > One note here - in the C++ broker, I'd like to pass a more restricted
> > > interface to Queue and Message to the messagestore.
> >
> > I agree; I'm going to be continuing with the dtx implementation on c++
> > and will do some refactoring as part of that, including the point you
> > mention here (e.g. some sort of PersistableQueue and PersistableMessage
> > interface).
> >
>
>

Re: [c++] Re: dtx API

Posted by Arnaud Simon <as...@redhat.com>.
+1

I first though about only sending bytes arrays and Uids around but it
was too restrictive so we went for what we currently have. 
I´ll synchronize with Gordon for defining PersistableQueue and
PersistableMessage interfaces.

Cheers

Arnaud


On Mon, 2007-03-26 at 14:42 +0100, Gordon Sim wrote:
> Alan Conway wrote:
> > One note here - in the C++ broker, I'd like to pass a more restricted 
> > interface to Queue and Message to the messagestore. 
> 
> I agree; I'm going to be continuing with the dtx implementation on c++ 
> and will do some refactoring as part of that, including the point you 
> mention here (e.g. some sort of PersistableQueue and PersistableMessage 
> interface).
> 


[c++] Re: dtx API

Posted by Gordon Sim <gs...@redhat.com>.
Alan Conway wrote:
> One note here - in the C++ broker, I'd like to pass a more restricted 
> interface to Queue and Message to the messagestore. 

I agree; I'm going to be continuing with the dtx implementation on c++ 
and will do some refactoring as part of that, including the point you 
mention here (e.g. some sort of PersistableQueue and PersistableMessage 
interface).


Re: dtx API

Posted by Alan Conway <ac...@redhat.com>.
Arnaud Simon wrote:
> Hi Rupert,
>
> It must have been filtered out. 
> So here it is in plain text: 
>
> Interface MessageStore
> {
> 	/**
> 	/* Create a queue.
> 	/*
> 	/* throws:
> 	/* InternalErrorException	
> 	/**
> 	void createQueue(Queue queue) throws 
> 						InternalErrorException
>   
One note here - in the C++ broker, I'd like to pass a more restricted 
interface to Queue and Message to the messagestore. The messagestore 
(and other future "plug-in" extensions I imagine) don't need access to 
the full internal definitions, and a more restricted interface would 
make future refactoring of internals much easier, especially if we do 
end up with more such plug-ins (e.g. 3rd party custom exchanges etc.) 
This came up in a practical sense for me during 0-9 refactoring - I had 
to read the whole bdbstore code to discover that it really only cares 
about being able to encode() and decode() queue and message objects and 
get unique identifiers from them. I don't know what the java interfaces 
are like but this might be worth considering - or it might be something 
for later.

Cheers,
Alan.

Re: dtx API

Posted by Arnaud Simon <as...@redhat.com>.
Hi Rupert,

It must have been filtered out. 
So here it is in plain text: 

Interface MessageStore
{
	/**
	/* Create a queue.
	/*
	/* throws:
	/* InternalErrorException	
	/**
	void createQueue(Queue queue) throws 
						InternalErrorException
									

	/**
	/* Destroy a queue.
	/*
	/* throws:
	/* InternalErrorException	
	/**
	void destroyQueue(Queue queue) throws 
						 InternalErrorException
						 QueueDoesntExistException

	/**
	/* stage a message. 
	/*
	/* throws:
	/* InternalErrorException	
	/**
	void stage(Message m) throws 
				    InternalErrorException

	/**
	/* Append more data with a previously staged message. 
	/*
	/* throws:
	/* InternalErrorException
	/* MessageDoesntExistException
	/**
	void appendContent(Message m, byte[] data) throws 
								 InternalErrorException
								 MessageDoesntExistException								
	

	/**
	/* load the content of previously staged or enqueued message. 
	/*
	/* throws:
	/* InternalErrorException
	/* MessageDoesntExistException
	/**
	void loadContent(Message m) throws 
				          InternalErrorException
					    MessageDoesntExistException		
			

	/**
	/* destroy a previously staged message identified by msgId.
	/*
	/* throws:
	/* InternalErrorException	
	/* MessageDoesntExistException
	/**
	void destroy(Message m) throws 
				      InternalErrorException
				      MessageDoesntExistException

	/**
	/* Enqueue a message under the scope of the transaction branch identified by xid when specified. Return the message identifier.                                          
	/*
	/* throws:
	/* InternalErrorException
	/* QueueDoesntExistException
	/* InvalidXidException
	/**
	Uid enqueue(Xid xid, Message m, Queue queue) throws 
								   InternalErrorException
								   QueueDoesntExistException
								   InvalidXidException
	
	/**
	/* Dequeue a message under the scope of the transaction branch identified by xid if specified.
	/*
	/* throws:
	/* InternalErrorException
	/* QueueDoesntExistException
	/* InvalidXidException
	/**
	void dequeue(Xid xid, Message m, Queue queue) throws 
								    InternalErrorException
								    QueueDoesntExistException
								    InvalidXidException	
	
	/*************************
	/* Recovery specific methods 
	/*************************

	/** 
	/* List all the persistent queues 
	/* Return a list of Queues
	/*
	/* throws:
	/* InternalErrorException
	/**
	List getAllQueues() throws InternalErrorException

	/**
	/* List all enqueued messages of a given queue 
	/*
	/* throws:
	/* InternalErrorException
	/* QueueDoesntExistException
	/**
	List getAllMessages(Queue queue) throws 
						   InternalErrorException
						   QueueDoesntExistException
}


Interface TransactionManager 
{
	/**
	/* Begin a transaction branch identified by Xid. 
	/*
	/* throws:                     
	/* InternalErrorException
	/* InvalidXidException
	/*
      /* possible outcomes are: 
	/* xa-ok: Normal execution.
      /* xa-rbrollback: The transaction branch was marked rollback-only for an unspecified reason.    
	/**      
  	int begin(Xid xid) throws                       
			       InternalErrorException
			       InvalidXidException

	/**
	/* Prepare the transaction branch identified by Xid
	/* 
	/* throws:
	/* InternalErrorException
      /* CommandInvalidException 
	/* UnknownXidException 
	/* possible outcomes are: 
	/* xa-ok: Normal execution.
      /* xa-rdonly: The transaction branch was read-only and has been committed.
      /* xa-rbrollback: The transaction branch was marked rollback-only for an unspecified reason.    
	/* xa-rbtimeout: The work represented by this transaction branch took too long.    
	/** 
	int prepare(Xid xid) throws                       
			     	   InternalErrorException
                       	   CommandInvalidException 
			         UnknownXidException                        
                
	/** 
	/* rollback the transaction branch identified by Xid
	/*
	/* throws                    
      /* InternalErrorException
      /* CommandInvalidException 
      /* UnknownXidException     
      /*   
	/* possible outcomes are:     
	/* xa-ok: Normal execution,
	/* xa-heurhaz: Due to some failure, the work done on behalf of the specified transaction branch may have been heuristically completed.
      /* xa-heurcom: Due to a heuristic decision, the work done on behalf of the specified transaction branch was committed.
	/* xa-heurrb: Due to a heuristic decision, the work done on behalf of the specified transaction branch was rolled back.
	/* xa-heurmix: Due to a heuristic decision, the work done on behalf of the specified transaction branch was partially committed and partially rolled back.
      /* xa-rbrollback: The broker marked the transaction branch rollback-only for an unspecified reason. 
	/* xa-rbtimeout: The work represented by this transaction branch took too long.      
	/**
      int rollback(Xid xid) throws                    
                        	    InternalErrorException
                        	    CommandInvalidException 
				    UnknownXidException     
              



	  /** 
	  /* Commit the transaction branch identified by Xid
	  /*
	  /* throws                    
        /* InternalErrorException
        /* CommandInvalidException 
        /* UnknownXidException     
        /* NotPreparedException
	  /* 
	  /* possible outcomes are: 
	  /* xa-ok: Normal execution,
	  /* xa-heurhaz: Due to some failure, the work done on behalf of the specified transaction branch may have been heuristically completed.
        /* xa-heurcom: Due to a heuristic decision, the work done on behalf of the specified transaction branch was committed.
	  /* xa-heurrb: Due to a heuristic decision, the work done on behalf of the specified transaction branch was rolled back.
	  /* xa-heurmix: Due to a heuristic decision, the work done on behalf of the specified transaction branch was partially committed and partially rolled back.
        /* xa-rbrollback: The broker marked the transaction branch rollback-only for an unspecified reason.	  
	  /* xa-rbtimeout: The work represented by this transaction branch took too long.    
        /**   
        int commit(Xid xid) throws 
				    InternalErrorException
        		          CommandInvalidException 
        			    UnknownXidException     
        			    NotPreparedException
               

	  /** 
	  /* One phase commit the transaction branch identified by Xid
	  /*
	  /* throws                    
        /* InternalErrorException
        /* CommandInvalidException 
        /* UnknownXidException            
	  /* 
	  /* possible outcomes are: 
	  /* xa-ok: Normal execution,
	  /* xa-heurhaz: Due to some failure, the work done on behalf of the specified transaction branch may have been heuristically completed.
        /* xa-heurcom: Due to a heuristic decision, the work done on behalf of the specified transaction branch was committed.
	  /* xa-heurrb: Due to a heuristic decision, the work done on behalf of the specified transaction branch was rolled back.
	  /* xa-heurmix: Due to a heuristic decision, the work done on behalf of the specified transaction branch was partially committed and partially rolled back.
        /* xa-rbrollback: The broker marked the transaction branch rollback-only for an unspecified reason.	  
	  /* xa-rbtimeout: The work represented by this transaction branch took too long.    
        /**   
        int commit_one_phase(Xid xid) throws 
					        InternalErrorException
        				        CommandInvalidException 
        				        UnknownXidException     
        	
	  
	  /** 
	  /* Forget about the transaction branch identified by Xid
	  /*
	  /* throws                    
        /* InternalErrorException
        /* CommandInvalidException 
        /* UnknownXidException       
	  /**
        void forget(Xid xid) throws 
			           InternalErrorException
        		           CommandInvalidException 
        		           UnknownXidException   

	  
	  /** 
	  /*  Sets the transaction branch timeout value in seconds.
	  /*
	  /* throws                    
        /* InternalErrorException        
        /* UnknownXidException    
	  /**   
	  void setTimeout(Xid xid, long timeout) throws 
				   	   		     InternalErrorException	        	          
        		           	 		     UnknownXidException   

	  /** 
	  /*  Gets the transaction branch timeout.
	  /*
	  /* throws                    
        /* InternalErrorException        
        /* UnknownXidException    
	  /**   
	  long getTimeout(Xid xid) throws 
				         InternalErrorException	        		          
        		               UnknownXidException   

	  /**
	  /* Get a list of Xids the RM has prepared or  heuristically completed.
	  /*
	  /* startscan: Indicates that recovery scan should start. 
	  /* endscan:   Indicates that the recovery scan should end after returning the Xids
	  /*
        /* throws                    
        /* InternalErrorException  
	  /* CommandInvalidException   
	  /**
	  List recover(boolean startscan, boolean endscan) throws 
						InternalErrorException   						 CommandInvalidException
}

Interface RecoveryManager 
{
	  /**
	  /* Check the status of all the transactions, rollback unprepared transactions and replay phase 2 of the commit protocol for transactions that are committing. Empty queues are also deleted. 
	  /* 
	  /* throws 
          /* InternalErrorException  
	  /**
	  void recoverFromFailure() throws 
			     		    InternalErrorException   

	  /** 
	  /* Periodic recovery of transactions that are not in progress. 
	  /* throws                    
        /* InternalErrorException  
	  /**
	  void periodicRecovery() throws 
			     		  InternalErrorException   
}

Interface Xid
{
 	  /**
	  /* Get the global transaction identifier.
    	  /*
    	  /**
        byte[] getGlobalTransactionId()

  	  /**
        /* Obtain the transaction branch identifier part of Xidas an array of bytes.
        /*
        /**
        byte[] getBranchQualifier()
 
        /**
        /* Obtain the format identifier part of the Xid.
        /*
        /**
        int getFormatId()
}

// Exceptions
InternalErrorException
CommandInvalidException 
UnknownXidException     
NotPreparedException
InvalidXidException
QueueDoesntExistException
MessageDoesntExistException

// Constants
Interface XAConstantes
{
// The rollback was caused by an unspecified reason.
int xa-rbrollback = 1
// A transaction branch took too long.    
int xa-rbtimeout = 2
// The transaction branch may have been heuristically completed.
Int xa-heurhaz = 3
// The transaction branch has been heuristically committed.
int Xa-heurcom = 4
// The transaction branch has been heuristically rolled back.
int xa-heurrb = 5
// The transaction branch has been heuristically committed and rolled back.
int xa-heurmix = 6
// The transaction branch was read-only and has been committed. 
int xa-rdonly =7 
// Normal execution.  
int xa-ok = 8 
}

On Mon, 2007-03-26 at 12:52 +0100, Rupert Smith wrote:
> Hi Arnaud,
> 
> There doesn't seem to be an attachment. Did it get filtered out by the
> mailing list?
> 
> Rupert
> 
> On 26/03/07, Arnaud Simon <as...@redhat.com> wrote:
>         Hello,
>         
>         I have attached with this email a proposal for APIs that I
>         expect to use
>         for dtx support. Those interfaces are the one I expect to
>         implement on
>         the Java broker. The C++ broker may use some slightly
>         different APIs but 
>         they should semantically be very similar.
>         Those APIs have been designed to:
>         1) make a clean distinction between transaction manager,
>         recovery
>         manager and persistence store.
>         2) Use whenever possible a direct mapping between dtx classes
>         and 
>         transaction manager methods. (you'll see that exceptions
>         should be
>         mapped to channel close with the corresponding error code and
>         results
>         can be returned straight ahead)
>         3) Use an unique way of identifying transactional work by
>         only 
>         propagating Xids all over the place. The underlying
>         implementation can
>         keep the mapping between its tx context representation and
>         Xids.
>         
>         I would very much appreciate your comments.
>         
>         For a synchronization purpose I am going to do the following
>         unless it 
>         clashes with somebody else's agenda:
>         - on the broker:
>         1) refactor the current code for supporting the proposed
>         interfaces
>         2) add dtx classes support
>         3) extend current dbd code for dtx support
>         4) add support for dtx around a file based store 
>         -on the client:
>         1) add dtx classes support
>         2) extend JMS implementation with XA capabilities
>         
>         I should start with the broker (1, 2 and 3) then with the
>         client and
>         finally do 4) on the broker.
>         
>         Regards 
>         
>         Arnaud
>         
> 


Re: dtx API

Posted by Rupert Smith <ru...@googlemail.com>.
Hi Arnaud,

There doesn't seem to be an attachment. Did it get filtered out by the
mailing list?

Rupert

On 26/03/07, Arnaud Simon <as...@redhat.com> wrote:
>
> Hello,
>
> I have attached with this email a proposal for APIs that I expect to use
> for dtx support. Those interfaces are the one I expect to implement on
> the Java broker. The C++ broker may use some slightly different APIs but
> they should semantically be very similar.
> Those APIs have been designed to:
> 1) make a clean distinction between transaction manager, recovery
> manager and persistence store.
> 2) Use whenever possible a direct mapping between dtx classes and
> transaction manager methods. (you'll see that exceptions should be
> mapped to channel close with the corresponding error code and results
> can be returned straight ahead)
> 3) Use an unique way of identifying transactional work by only
> propagating Xids all over the place. The underlying implementation can
> keep the mapping between its tx context representation and Xids.
>
> I would very much appreciate your comments.
>
> For a synchronization purpose I am going to do the following unless it
> clashes with somebody else's agenda:
> - on the broker:
> 1) refactor the current code for supporting the proposed interfaces
> 2) add dtx classes support
> 3) extend current dbd code for dtx support
> 4) add support for dtx around a file based store
> -on the client:
> 1) add dtx classes support
> 2) extend JMS implementation with XA capabilities
>
> I should start with the broker (1, 2 and 3) then with the client and
> finally do 4) on the broker.
>
> Regards
>
> Arnaud
>
>