You are viewing a plain text version of this content. The canonical link for it is here.
Posted to sandesha-dev@ws.apache.org by Amila Suriarachchi <am...@gmail.com> on 2008/10/24 11:26:09 UTC
Sandesha2 synchronization and dead lock handling.
hi all,
This is regarding the issue [1].
First of all as I learned Sandesha2 uses different beans to keep the state
of the sequence and the messages. In a dual channel mode
different threads can access these beans and update them concurrently. So
the synchronization of these beans done by using the
storage level transactions. Therefore Sandesha2 needs an storage which
supports isolated transactions.
To synchronize these beans the transactions must be completely isolated. i.e
It should not allow simultaneous reads of
same record from different transactions. Therefore I think the problem I saw
on[1] because not isolating the transactions properly.
Then I increased the transaction isolation to fix the above problem. It
fixed that problem but results in dead locks.
The reason I believe for this dead locks is that different transactions try
to access the data base tables in different order.
But unfortunately I could not fix the issue.
Normally these types of dead locks are prevented by accessing resources in
same order. Does Sandesha2 follows such a order or any
other technique?
Or is there any other reason for this dead locks and synchronization
problems? Can someone
have a better idea of Sandesha2 Design shed some light on this?
thanks,
Amila.
[1] http://issues.apache.org/jira/browse/SANDESHA2-179
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Re: Sandesha2 synchronization and dead lock handling.
Posted by Amila Suriarachchi <am...@gmail.com>.
On Fri, Oct 31, 2008 at 5:19 PM, Andrew K Gatford <GA...@uk.ibm.com>wrote:
> The Sandesha2 layer was gradually changed to support this structure. It
> was always then left to the storage managers to ensure that the beans were
> sufficiently locked when returned to the sandesha2 code.
yes, this is what I was doing as well. But what I found was that still
there are a lot of such paths, when tested with the method Thomas has
suggested. Then there is a problem of that whether these are significant in
real practical usage.
thanks,
Amila.
>
> I think one of the last times that I've had to fix a dead lock was
> probably in July last year ->
>
> Revision: 557671
> Author: gatfora
> Date: 16:12:20, 19 July 2007
> Message:
> Fix deadlock when piggybacking Acks where the SenderBean and RmdBean locks
> are taken in the wrong order
> ----
> Modified :
>
> /webservices/sandesha/trunk/java/modules/core/src/main/java/org/apache/sandesha2/util/AcknowledgementManager.java
> Modified :
>
> /webservices/sandesha/trunk/java/modules/core/src/main/java/org/apache/sandesha2/workers/SenderWorker.java
>
> I might have fixed some later than this that might have been fixed under
> another change.
>
> Andrew Gatford
> Technical Project Lead
> Websphere ESB Foundation Technologies
> Hursley MP211
> IBM United Kingdom Laboratories, Hursley Park, Winchester, SO21 2JN
> Telephone :
> Internal (7) 245743
> External 01962 815743
> Internet : gatfora@uk.ibm.com
>
>
>
> From:
> "Amila Suriarachchi" <am...@gmail.com>
> To:
> Andrew K Gatford/UK/IBM@IBMGB
> Cc:
> "sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
> Date:
> 31/10/2008 05:57
> Subject:
> Re: Sandesha2 synchronization and dead lock handling.
>
>
>
>
>
> On Fri, Oct 24, 2008 at 3:31 PM, Andrew K Gatford <GA...@uk.ibm.com>
> wrote:
> I went through similar pain when implementing a StorageManager and
> encountered a number of deadlocks similar to the ones that you describe.
> What I have gradually done is eliminate these in both the InMemory store
> and my store by changing the ordering the beans were taken in.
>
> In general the beans are taken in this order.
>
> RMSBean or RMDBean followed by
> SenderBean or InvokerBean.
>
> Did you do this at the storage level or at Sandesha2 level. Could you
> please suggest the way to implement this
> with the current jdbc persistence storage?
>
> thanks,
> Amila.
>
>
> In cases where both the RMSBean and RMDBean are locked, they tend to be
> taken in that order - RMS followed by RMD.
> The one thing that I do know is that it is fairly easy to introduce new
> deadlocks by slightly altering the order that beans are read.
>
> The one question I have is how does the jdbc store handle multiple threads
> accessing multiple sequences, or even a single sequence, but with multiple
> threads sending multiple requests. From my experience this is where we
> have found a lot of problems in the InMemory store and I expect to be even
> more painful with a jdbc store.
>
> Andrew Gatford
> Technical Project Lead
> Websphere ESB Foundation Technologies
> Hursley MP211
> IBM United Kingdom Laboratories, Hursley Park, Winchester, SO21 2JN
> Telephone :
> Internal (7) 245743
> External 01962 815743
> Internet : gatfora@uk.ibm.com
>
>
>
> From:
> "Amila Suriarachchi" <am...@gmail.com>
> To:
> "sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
> Date:
> 24/10/2008 10:30
> Subject:
> Sandesha2 synchronization and dead lock handling.
>
>
>
> hi all,
>
> This is regarding the issue [1].
>
> First of all as I learned Sandesha2 uses different beans to keep the state
> of the sequence and the messages. In a dual channel mode
> different threads can access these beans and update them concurrently. So
> the synchronization of these beans done by using the
> storage level transactions. Therefore Sandesha2 needs an storage which
> supports isolated transactions.
>
> To synchronize these beans the transactions must be completely isolated.
> i.e It should not allow simultaneous reads of
> same record from different transactions. Therefore I think the problem I
> saw on[1] because not isolating the transactions properly.
>
> Then I increased the transaction isolation to fix the above problem. It
> fixed that problem but results in dead locks.
> The reason I believe for this dead locks is that different transactions
> try to access the data base tables in different order.
> But unfortunately I could not fix the issue.
>
> Normally these types of dead locks are prevented by accessing resources in
> same order. Does Sandesha2 follows such a order or any
> other technique?
>
> Or is there any other reason for this dead locks and synchronization
> problems? Can someone
> have a better idea of Sandesha2 Design shed some light on this?
>
> thanks,
> Amila.
>
>
> [1] http://issues.apache.org/jira/browse/SANDESHA2-179
> --
> Amila Suriarachchi
> WSO2 Inc.
> blog: http://amilachinthaka.blogspot.com/
>
>
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
>
>
>
>
>
>
>
> --
> Amila Suriarachchi
> WSO2 Inc.
> blog: http://amilachinthaka.blogspot.com/
>
>
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
>
>
>
>
>
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Re: Sandesha2 synchronization and dead lock handling.
Posted by Andrew K Gatford <GA...@uk.ibm.com>.
The Sandesha2 layer was gradually changed to support this structure. It
was always then left to the storage managers to ensure that the beans were
sufficiently locked when returned to the sandesha2 code.
I think one of the last times that I've had to fix a dead lock was
probably in July last year ->
Revision: 557671
Author: gatfora
Date: 16:12:20, 19 July 2007
Message:
Fix deadlock when piggybacking Acks where the SenderBean and RmdBean locks
are taken in the wrong order
----
Modified :
/webservices/sandesha/trunk/java/modules/core/src/main/java/org/apache/sandesha2/util/AcknowledgementManager.java
Modified :
/webservices/sandesha/trunk/java/modules/core/src/main/java/org/apache/sandesha2/workers/SenderWorker.java
I might have fixed some later than this that might have been fixed under
another change.
Andrew Gatford
Technical Project Lead
Websphere ESB Foundation Technologies
Hursley MP211
IBM United Kingdom Laboratories, Hursley Park, Winchester, SO21 2JN
Telephone :
Internal (7) 245743
External 01962 815743
Internet : gatfora@uk.ibm.com
From:
"Amila Suriarachchi" <am...@gmail.com>
To:
Andrew K Gatford/UK/IBM@IBMGB
Cc:
"sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
Date:
31/10/2008 05:57
Subject:
Re: Sandesha2 synchronization and dead lock handling.
On Fri, Oct 24, 2008 at 3:31 PM, Andrew K Gatford <GA...@uk.ibm.com>
wrote:
I went through similar pain when implementing a StorageManager and
encountered a number of deadlocks similar to the ones that you describe.
What I have gradually done is eliminate these in both the InMemory store
and my store by changing the ordering the beans were taken in.
In general the beans are taken in this order.
RMSBean or RMDBean followed by
SenderBean or InvokerBean.
Did you do this at the storage level or at Sandesha2 level. Could you
please suggest the way to implement this
with the current jdbc persistence storage?
thanks,
Amila.
In cases where both the RMSBean and RMDBean are locked, they tend to be
taken in that order - RMS followed by RMD.
The one thing that I do know is that it is fairly easy to introduce new
deadlocks by slightly altering the order that beans are read.
The one question I have is how does the jdbc store handle multiple threads
accessing multiple sequences, or even a single sequence, but with multiple
threads sending multiple requests. From my experience this is where we
have found a lot of problems in the InMemory store and I expect to be even
more painful with a jdbc store.
Andrew Gatford
Technical Project Lead
Websphere ESB Foundation Technologies
Hursley MP211
IBM United Kingdom Laboratories, Hursley Park, Winchester, SO21 2JN
Telephone :
Internal (7) 245743
External 01962 815743
Internet : gatfora@uk.ibm.com
From:
"Amila Suriarachchi" <am...@gmail.com>
To:
"sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
Date:
24/10/2008 10:30
Subject:
Sandesha2 synchronization and dead lock handling.
hi all,
This is regarding the issue [1].
First of all as I learned Sandesha2 uses different beans to keep the state
of the sequence and the messages. In a dual channel mode
different threads can access these beans and update them concurrently. So
the synchronization of these beans done by using the
storage level transactions. Therefore Sandesha2 needs an storage which
supports isolated transactions.
To synchronize these beans the transactions must be completely isolated.
i.e It should not allow simultaneous reads of
same record from different transactions. Therefore I think the problem I
saw on[1] because not isolating the transactions properly.
Then I increased the transaction isolation to fix the above problem. It
fixed that problem but results in dead locks.
The reason I believe for this dead locks is that different transactions
try to access the data base tables in different order.
But unfortunately I could not fix the issue.
Normally these types of dead locks are prevented by accessing resources in
same order. Does Sandesha2 follows such a order or any
other technique?
Or is there any other reason for this dead locks and synchronization
problems? Can someone
have a better idea of Sandesha2 Design shed some light on this?
thanks,
Amila.
[1] http://issues.apache.org/jira/browse/SANDESHA2-179
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
---------------------------------------------------------------------
To unsubscribe, e-mail: sandesha-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: sandesha-dev-help@ws.apache.org
Re: Sandesha2 synchronization and dead lock handling.
Posted by Amila Suriarachchi <am...@gmail.com>.
On Fri, Oct 24, 2008 at 3:31 PM, Andrew K Gatford <GA...@uk.ibm.com>wrote:
> I went through similar pain when implementing a StorageManager and
> encountered a number of deadlocks similar to the ones that you describe.
> What I have gradually done is eliminate these in both the InMemory store
> and my store by changing the ordering the beans were taken in.
>
> In general the beans are taken in this order.
>
> RMSBean or RMDBean followed by
> SenderBean or InvokerBean.
Did you do this at the storage level or at Sandesha2 level. Could you please
suggest the way to implement this
with the current jdbc persistence storage?
thanks,
Amila.
>
>
> In cases where both the RMSBean and RMDBean are locked, they tend to be
> taken in that order - RMS followed by RMD.
> The one thing that I do know is that it is fairly easy to introduce new
> deadlocks by slightly altering the order that beans are read.
>
> The one question I have is how does the jdbc store handle multiple threads
> accessing multiple sequences, or even a single sequence, but with multiple
> threads sending multiple requests. From my experience this is where we
> have found a lot of problems in the InMemory store and I expect to be even
> more painful with a jdbc store.
>
> Andrew Gatford
> Technical Project Lead
> Websphere ESB Foundation Technologies
> Hursley MP211
> IBM United Kingdom Laboratories, Hursley Park, Winchester, SO21 2JN
> Telephone :
> Internal (7) 245743
> External 01962 815743
> Internet : gatfora@uk.ibm.com
>
>
>
> From:
> "Amila Suriarachchi" <am...@gmail.com>
> To:
> "sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
> Date:
> 24/10/2008 10:30
> Subject:
> Sandesha2 synchronization and dead lock handling.
>
>
>
> hi all,
>
> This is regarding the issue [1].
>
> First of all as I learned Sandesha2 uses different beans to keep the state
> of the sequence and the messages. In a dual channel mode
> different threads can access these beans and update them concurrently. So
> the synchronization of these beans done by using the
> storage level transactions. Therefore Sandesha2 needs an storage which
> supports isolated transactions.
>
> To synchronize these beans the transactions must be completely isolated.
> i.e It should not allow simultaneous reads of
> same record from different transactions. Therefore I think the problem I
> saw on[1] because not isolating the transactions properly.
>
> Then I increased the transaction isolation to fix the above problem. It
> fixed that problem but results in dead locks.
> The reason I believe for this dead locks is that different transactions
> try to access the data base tables in different order.
> But unfortunately I could not fix the issue.
>
> Normally these types of dead locks are prevented by accessing resources in
> same order. Does Sandesha2 follows such a order or any
> other technique?
>
> Or is there any other reason for this dead locks and synchronization
> problems? Can someone
> have a better idea of Sandesha2 Design shed some light on this?
>
> thanks,
> Amila.
>
>
> [1] http://issues.apache.org/jira/browse/SANDESHA2-179
> --
> Amila Suriarachchi
> WSO2 Inc.
> blog: http://amilachinthaka.blogspot.com/
>
>
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
>
>
>
>
>
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Re: Sandesha2 synchronization and dead lock handling.
Posted by Andrew K Gatford <GA...@uk.ibm.com>.
That may work - there are a couple of possible problems that I can see.
For example - if you have a store that is shared with a number of servers
and the sequence information is shared for all those servers, having an
independent lock manager would have to be able to cope with negotiating
with all those servers.
The second possible problem is that even with the independent lock manager
- depending on the DB and the isolation levels etc, it is possible for
lock manager to grant a lock, but the DB to fail because it has a table
locked, or a row locked. This is why I was interested in what happens
when running with multiple sequences and using the jdbc store as it could
suffer from that sort of problem. Maybe I'm being too pessimistic -
perhaps it will all just work :)
Andrew Gatford
Technical Project Lead
Websphere ESB Foundation Technologies
Hursley MP211
IBM United Kingdom Laboratories, Hursley Park, Winchester, SO21 2JN
Telephone :
Internal (7) 245743
External 01962 815743
Internet : gatfora@uk.ibm.com
From:
Thomas McKiernan/UK/IBM@IBMGB
To:
Andrew K Gatford/UK/IBM@IBMGB
Cc:
"Amila Suriarachchi" <am...@gmail.com>,
"sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
Date:
24/10/2008 11:36
Subject:
Re: Sandesha2 synchronization and dead lock handling.
How about a lock manager impl independent of any particular store's impl.
It could be abstract if necessary.
Basically, this has a hierarchy of classes (beans) hard coded.
If you use a store to access a bean then the store impl's tran calls into
the independent lock manager.
Any attempt to enlist outside of the locking hierarchy results in a hard
runtime error and a rollback of the tran.
Is this too naive?
----------------------------------
Thomas McKiernan
WebSphere Messaging Development,
IBM United Kingdom Limited
Internal Phone: 248241
External Phone: +44 (0)1962 818241
Mobile: +44 (0)789 1737497
Email: MCKIERNA@uk.ibm.com
Mail Point 211, IBM, Hursley Park, Winchester, Hampshire, England, SO21
2JN
Caminante, no hay camino
Se hace camino al andar.
("Walker, there is no path; the path is made by walking.") Antonio
Machado
From:
Andrew K Gatford/UK/IBM@IBMGB
To:
"Amila Suriarachchi" <am...@gmail.com>
Cc:
"sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
Date:
24/10/2008 11:07
Subject:
Re: Sandesha2 synchronization and dead lock handling.
I went through similar pain when implementing a StorageManager and
encountered a number of deadlocks similar to the ones that you describe.
What I have gradually done is eliminate these in both the InMemory store
and my store by changing the ordering the beans were taken in.
In general the beans are taken in this order.
RMSBean or RMDBean followed by
SenderBean or InvokerBean.
In cases where both the RMSBean and RMDBean are locked, they tend to be
taken in that order - RMS followed by RMD.
The one thing that I do know is that it is fairly easy to introduce new
deadlocks by slightly altering the order that beans are read.
The one question I have is how does the jdbc store handle multiple threads
accessing multiple sequences, or even a single sequence, but with multiple
threads sending multiple requests. From my experience this is where we
have found a lot of problems in the InMemory store and I expect to be even
more painful with a jdbc store.
Andrew Gatford
Technical Project Lead
Websphere ESB Foundation Technologies
Hursley MP211
IBM United Kingdom Laboratories, Hursley Park, Winchester, SO21 2JN
Telephone :
Internal (7) 245743
External 01962 815743
Internet : gatfora@uk.ibm.com
From:
"Amila Suriarachchi" <am...@gmail.com>
To:
"sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
Date:
24/10/2008 10:30
Subject:
Sandesha2 synchronization and dead lock handling.
hi all,
This is regarding the issue [1].
First of all as I learned Sandesha2 uses different beans to keep the state
of the sequence and the messages. In a dual channel mode
different threads can access these beans and update them concurrently. So
the synchronization of these beans done by using the
storage level transactions. Therefore Sandesha2 needs an storage which
supports isolated transactions.
To synchronize these beans the transactions must be completely isolated.
i.e It should not allow simultaneous reads of
same record from different transactions. Therefore I think the problem I
saw on[1] because not isolating the transactions properly.
Then I increased the transaction isolation to fix the above problem. It
fixed that problem but results in dead locks.
The reason I believe for this dead locks is that different transactions
try to access the data base tables in different order.
But unfortunately I could not fix the issue.
Normally these types of dead locks are prevented by accessing resources in
same order. Does Sandesha2 follows such a order or any
other technique?
Or is there any other reason for this dead locks and synchronization
problems? Can someone
have a better idea of Sandesha2 Design shed some light on this?
thanks,
Amila.
[1] http://issues.apache.org/jira/browse/SANDESHA2-179
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
---------------------------------------------------------------------
To unsubscribe, e-mail: sandesha-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: sandesha-dev-help@ws.apache.org
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
---------------------------------------------------------------------
To unsubscribe, e-mail: sandesha-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: sandesha-dev-help@ws.apache.org
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
---------------------------------------------------------------------
To unsubscribe, e-mail: sandesha-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: sandesha-dev-help@ws.apache.org
RE: Sandesha2 synchronization and dead lock handling.
Posted by "Lucente, Richard D" <ri...@lmco.com>.
If it's helpful, my company had some difficulty completing a persistence
implementation for sandesha2-1.2. What we observed was the inmemory
solution appears to enforce pessimistic row locking and the serializable
transaction isolation level. Once beans are enlisted in a transaction,
all other threads attempting to access the bean block until the bean is
released from its transaction. It's not possible to implement a
persistence solution without combing through the inmemory
implementations.
From: Amila Suriarachchi [mailto:amilasuriarachchi@gmail.com]
Sent: Friday, October 24, 2008 9:04 AM
To: Thomas McKiernan
Cc: Andrew K Gatford; sandesha-dev@ws.apache.org
Subject: Re: Sandesha2 synchronization and dead lock handling.
On Fri, Oct 24, 2008 at 3:55 PM, Thomas McKiernan <MC...@uk.ibm.com>
wrote:
How about a lock manager impl independent of any particular store's
impl.
It could be abstract if necessary.
Basically, this has a hierarchy of classes (beans) hard coded.
If you use a store to access a bean then the store impl's tran calls
into
the independent lock manager.
I feel this is a kind hack for the problem. And also as Andrew has
mentioned this won't work in a
distributed environment.
For me the correct solution is to go through all the transactions and
make an order of which
transactions access the beans. But apparently this is also seems to be
difficult since a lot
of transactions has start and commits.
So have to think bit more.
thanks,
Amila.
Any attempt to enlist outside of the locking hierarchy results
in a hard
runtime error and a rollback of the tran.
Is this too naive?
----------------------------------
Thomas McKiernan
WebSphere Messaging Development,
IBM United Kingdom Limited
Internal Phone: 248241
External Phone: +44 (0)1962 818241
Mobile: +44 (0)789 1737497
Email: MCKIERNA@uk.ibm.com
Mail Point 211, IBM, Hursley Park, Winchester, Hampshire,
England, SO21
2JN
Caminante, no hay camino
Se hace camino al andar.
("Walker, there is no path; the path is made by walking.")
Antonio
Machado
From:
Andrew K Gatford/UK/IBM@IBMGB
To:
"Amila Suriarachchi" <am...@gmail.com>
Cc:
"sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
Date:
24/10/2008 11:07
Subject:
Re: Sandesha2 synchronization and dead lock handling.
I went through similar pain when implementing a StorageManager
and
encountered a number of deadlocks similar to the ones that you
describe.
What I have gradually done is eliminate these in both the
InMemory store
and my store by changing the ordering the beans were taken in.
In general the beans are taken in this order.
RMSBean or RMDBean followed by
SenderBean or InvokerBean.
In cases where both the RMSBean and RMDBean are locked, they
tend to be
taken in that order - RMS followed by RMD.
The one thing that I do know is that it is fairly easy to
introduce new
deadlocks by slightly altering the order that beans are read.
The one question I have is how does the jdbc store handle
multiple threads
accessing multiple sequences, or even a single sequence, but
with multiple
threads sending multiple requests. From my experience this is
where we
have found a lot of problems in the InMemory store and I expect
to be even
more painful with a jdbc store.
Andrew Gatford
Technical Project Lead
Websphere ESB Foundation Technologies
Hursley MP211
IBM United Kingdom Laboratories, Hursley Park, Winchester, SO21
2JN
Telephone :
Internal (7) 245743
External 01962 815743
Internet : gatfora@uk.ibm.com
From:
"Amila Suriarachchi" <am...@gmail.com>
To:
"sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
Date:
24/10/2008 10:30
Subject:
Sandesha2 synchronization and dead lock handling.
hi all,
This is regarding the issue [1].
First of all as I learned Sandesha2 uses different beans to keep
the state
of the sequence and the messages. In a dual channel mode
different threads can access these beans and update them
concurrently. So
the synchronization of these beans done by using the
storage level transactions. Therefore Sandesha2 needs an storage
which
supports isolated transactions.
To synchronize these beans the transactions must be completely
isolated.
i.e It should not allow simultaneous reads of
same record from different transactions. Therefore I think the
problem I
saw on[1] because not isolating the transactions properly.
Then I increased the transaction isolation to fix the above
problem. It
fixed that problem but results in dead locks.
The reason I believe for this dead locks is that different
transactions
try to access the data base tables in different order.
But unfortunately I could not fix the issue.
Normally these types of dead locks are prevented by accessing
resources in
same order. Does Sandesha2 follows such a order or any
other technique?
Or is there any other reason for this dead locks and
synchronization
problems? Can someone
have a better idea of Sandesha2 Design shed some light on this?
thanks,
Amila.
[1] http://issues.apache.org/jira/browse/SANDESHA2-179
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales
with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth,
Hampshire PO6 3AU
---------------------------------------------------------------------
To unsubscribe, e-mail: sandesha-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: sandesha-dev-help@ws.apache.org
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales
with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth,
Hampshire PO6 3AU
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Re: Sandesha2 synchronization and dead lock handling.
Posted by Amila Suriarachchi <am...@gmail.com>.
On Tue, Oct 28, 2008 at 11:26 AM, Amila Suriarachchi <
amilasuriarachchi@gmail.com> wrote:
>
>
> On Fri, Oct 24, 2008 at 7:54 PM, Thomas McKiernan <MC...@uk.ibm.com>wrote:
>
>> I think Andrew was saying that there is already an implicit order for the
>> beans.
>
>
> yes this is correct. most of the transactions follows a same order except
> some transactions.
> What about making an explicit order so that we can change the code whenever
> found an exception.
>
> we can have an order like RMSBean, RMDBean, MessageBean, SenderBean.
>
>
>> The point is people keep changing the code and breaking that order.
>> Therefore this is not quite a hack but an attempt to make explicit what is
>> already implied in the open src code.
>>
>> It would not work for the distributed case but, assuming each distributed
>> thread was behaving correctly i.e. obeying the correct locking order then
>> this would not be an issue.
>> And this would catch any thread disobeying that ordering.
>
>
Please see the attached patch I have added some similar thing as you have
mentioned. Here I could not locate the InvokerBean.
Then I proceed with this change. I found a lot of places where it does not
follow the given order. In order to proceed I put another set of methods
called locks which are called earlier to make the sequence correct.
In this way I found a lot of places which does not follow this order. Of
course this may not be the correct order to follow.
Based on the above observations we can possibly have following options to
address this problem.
1. Although there are dead locks sequence get finished. I think this is the
way Sandesha2 is written. if there is an error it can roll back and proceed.
But lot of people may not like it.
2. To implement a possible deadlock detection system (As suggested above )
and remove them either using some dummy locks or reformatting the code. Here
reformatting the code may not feasible all the time.
Any thoughts?
thanks,
Amila.
>
> As I understood in your method what you try to do is to have an explicit
> order of accessing the
> beans and raise an exception if one transaction make an exception.
>
> In this case also don't we have to go and change the transaction to have it
> correct order (the explicit order you have define). Otherwise it keep on
> failing.
>
> But this will certainly improve the transaction debugging.
>
> thanks,
> Amila.
>
>
>
>>
>>
>> ----------------------------------
>> Thomas McKiernan
>>
>> WebSphere Messaging Development,
>> IBM United Kingdom Limited
>>
>> Internal Phone: 248241
>> External Phone: +44 (0)1962 818241
>> Mobile: +44 (0)789 1737497
>> Email: MCKIERNA@uk.ibm.com
>>
>> Mail Point 211, IBM, Hursley Park, Winchester, Hampshire, England, SO21
>> 2JN
>>
>>
>> Caminante, no hay camino
>> Se hace camino al andar.
>> ("Walker, there is no path; the path is made by walking.") Antonio
>> Machado
>>
>>
>>
>> From:
>> "Amila Suriarachchi" <am...@gmail.com>
>> To:
>> Thomas McKiernan/UK/IBM@IBMGB
>> Cc:
>> Andrew K Gatford/UK/IBM@IBMGB, "sandesha-dev@ws.apache.org"
>> <sa...@ws.apache.org>
>> Date:
>> 24/10/2008 14:06
>> Subject:
>> Re: Sandesha2 synchronization and dead lock handling.
>>
>>
>>
>>
>>
>> On Fri, Oct 24, 2008 at 3:55 PM, Thomas McKiernan <MC...@uk.ibm.com>
>> wrote:
>> How about a lock manager impl independent of any particular store's impl.
>> It could be abstract if necessary.
>>
>> Basically, this has a hierarchy of classes (beans) hard coded.
>> If you use a store to access a bean then the store impl's tran calls into
>> the independent lock manager.
>>
>> I feel this is a kind hack for the problem. And also as Andrew has
>> mentioned this won't work in a
>> distributed environment.
>> For me the correct solution is to go through all the transactions and make
>> an order of which
>> transactions access the beans. But apparently this is also seems to be
>> difficult since a lot
>> of transactions has start and commits.
>> So have to think bit more.
>>
>> thanks,
>> Amila.
>>
>>
>>
>>
>> Any attempt to enlist outside of the locking hierarchy results in a hard
>> runtime error and a rollback of the tran.
>>
>> Is this too naive?
>>
>> ----------------------------------
>> Thomas McKiernan
>>
>> WebSphere Messaging Development,
>> IBM United Kingdom Limited
>>
>> Internal Phone: 248241
>> External Phone: +44 (0)1962 818241
>> Mobile: +44 (0)789 1737497
>> Email: MCKIERNA@uk.ibm.com
>>
>> Mail Point 211, IBM, Hursley Park, Winchester, Hampshire, England, SO21
>> 2JN
>>
>>
>> Caminante, no hay camino
>> Se hace camino al andar.
>> ("Walker, there is no path; the path is made by walking.") Antonio
>> Machado
>>
>>
>>
>> From:
>> Andrew K Gatford/UK/IBM@IBMGB
>> To:
>> "Amila Suriarachchi" <am...@gmail.com>
>> Cc:
>> "sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
>> Date:
>> 24/10/2008 11:07
>> Subject:
>> Re: Sandesha2 synchronization and dead lock handling.
>>
>>
>>
>> I went through similar pain when implementing a StorageManager and
>> encountered a number of deadlocks similar to the ones that you describe.
>> What I have gradually done is eliminate these in both the InMemory store
>> and my store by changing the ordering the beans were taken in.
>>
>> In general the beans are taken in this order.
>>
>> RMSBean or RMDBean followed by
>> SenderBean or InvokerBean.
>>
>> In cases where both the RMSBean and RMDBean are locked, they tend to be
>> taken in that order - RMS followed by RMD.
>> The one thing that I do know is that it is fairly easy to introduce new
>> deadlocks by slightly altering the order that beans are read.
>>
>> The one question I have is how does the jdbc store handle multiple threads
>>
>> accessing multiple sequences, or even a single sequence, but with multiple
>>
>> threads sending multiple requests. From my experience this is where we
>> have found a lot of problems in the InMemory store and I expect to be even
>>
>> more painful with a jdbc store.
>>
>> Andrew Gatford
>> Technical Project Lead
>> Websphere ESB Foundation Technologies
>> Hursley MP211
>> IBM United Kingdom Laboratories, Hursley Park, Winchester, SO21 2JN
>> Telephone :
>> Internal (7) 245743
>> External 01962 815743
>> Internet : gatfora@uk.ibm.com
>>
>>
>>
>> From:
>> "Amila Suriarachchi" <am...@gmail.com>
>> To:
>> "sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
>> Date:
>> 24/10/2008 10:30
>> Subject:
>> Sandesha2 synchronization and dead lock handling.
>>
>>
>>
>> hi all,
>>
>> This is regarding the issue [1].
>>
>> First of all as I learned Sandesha2 uses different beans to keep the state
>>
>> of the sequence and the messages. In a dual channel mode
>> different threads can access these beans and update them concurrently. So
>> the synchronization of these beans done by using the
>> storage level transactions. Therefore Sandesha2 needs an storage which
>> supports isolated transactions.
>>
>> To synchronize these beans the transactions must be completely isolated.
>> i.e It should not allow simultaneous reads of
>> same record from different transactions. Therefore I think the problem I
>> saw on[1] because not isolating the transactions properly.
>>
>> Then I increased the transaction isolation to fix the above problem. It
>> fixed that problem but results in dead locks.
>> The reason I believe for this dead locks is that different transactions
>> try to access the data base tables in different order.
>> But unfortunately I could not fix the issue.
>>
>> Normally these types of dead locks are prevented by accessing resources in
>>
>> same order. Does Sandesha2 follows such a order or any
>> other technique?
>>
>> Or is there any other reason for this dead locks and synchronization
>> problems? Can someone
>> have a better idea of Sandesha2 Design shed some light on this?
>>
>> thanks,
>> Amila.
>>
>>
>> [1] http://issues.apache.org/jira/browse/SANDESHA2-179
>> --
>> Amila Suriarachchi
>> WSO2 Inc.
>> blog: http://amilachinthaka.blogspot.com/
>>
>>
>>
>>
>>
>>
>> Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with number
>> 741598.
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: sandesha-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: sandesha-dev-help@ws.apache.org
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with number
>> 741598.
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Amila Suriarachchi
>> WSO2 Inc.
>> blog: http://amilachinthaka.blogspot.com/
>>
>>
>>
>>
>>
>>
>>
>> Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with number
>> 741598.
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> Amila Suriarachchi
> WSO2 Inc.
> blog: http://amilachinthaka.blogspot.com/
>
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Re: Sandesha2 synchronization and dead lock handling.
Posted by Amila Suriarachchi <am...@gmail.com>.
On Fri, Oct 24, 2008 at 7:54 PM, Thomas McKiernan <MC...@uk.ibm.com>wrote:
> I think Andrew was saying that there is already an implicit order for the
> beans.
yes this is correct. most of the transactions follows a same order except
some transactions.
What about making an explicit order so that we can change the code whenever
found an exception.
we can have an order like RMSBean, RMDBean, MessageBean, SenderBean.
> The point is people keep changing the code and breaking that order.
> Therefore this is not quite a hack but an attempt to make explicit what is
> already implied in the open src code.
>
> It would not work for the distributed case but, assuming each distributed
> thread was behaving correctly i.e. obeying the correct locking order then
> this would not be an issue.
> And this would catch any thread disobeying that ordering.
As I understood in your method what you try to do is to have an explicit
order of accessing the
beans and raise an exception if one transaction make an exception.
In this case also don't we have to go and change the transaction to have it
correct order (the explicit order you have define). Otherwise it keep on
failing.
But this will certainly improve the transaction debugging.
thanks,
Amila.
>
>
> ----------------------------------
> Thomas McKiernan
>
> WebSphere Messaging Development,
> IBM United Kingdom Limited
>
> Internal Phone: 248241
> External Phone: +44 (0)1962 818241
> Mobile: +44 (0)789 1737497
> Email: MCKIERNA@uk.ibm.com
>
> Mail Point 211, IBM, Hursley Park, Winchester, Hampshire, England, SO21
> 2JN
>
>
> Caminante, no hay camino
> Se hace camino al andar.
> ("Walker, there is no path; the path is made by walking.") Antonio
> Machado
>
>
>
> From:
> "Amila Suriarachchi" <am...@gmail.com>
> To:
> Thomas McKiernan/UK/IBM@IBMGB
> Cc:
> Andrew K Gatford/UK/IBM@IBMGB, "sandesha-dev@ws.apache.org"
> <sa...@ws.apache.org>
> Date:
> 24/10/2008 14:06
> Subject:
> Re: Sandesha2 synchronization and dead lock handling.
>
>
>
>
>
> On Fri, Oct 24, 2008 at 3:55 PM, Thomas McKiernan <MC...@uk.ibm.com>
> wrote:
> How about a lock manager impl independent of any particular store's impl.
> It could be abstract if necessary.
>
> Basically, this has a hierarchy of classes (beans) hard coded.
> If you use a store to access a bean then the store impl's tran calls into
> the independent lock manager.
>
> I feel this is a kind hack for the problem. And also as Andrew has
> mentioned this won't work in a
> distributed environment.
> For me the correct solution is to go through all the transactions and make
> an order of which
> transactions access the beans. But apparently this is also seems to be
> difficult since a lot
> of transactions has start and commits.
> So have to think bit more.
>
> thanks,
> Amila.
>
>
>
>
> Any attempt to enlist outside of the locking hierarchy results in a hard
> runtime error and a rollback of the tran.
>
> Is this too naive?
>
> ----------------------------------
> Thomas McKiernan
>
> WebSphere Messaging Development,
> IBM United Kingdom Limited
>
> Internal Phone: 248241
> External Phone: +44 (0)1962 818241
> Mobile: +44 (0)789 1737497
> Email: MCKIERNA@uk.ibm.com
>
> Mail Point 211, IBM, Hursley Park, Winchester, Hampshire, England, SO21
> 2JN
>
>
> Caminante, no hay camino
> Se hace camino al andar.
> ("Walker, there is no path; the path is made by walking.") Antonio
> Machado
>
>
>
> From:
> Andrew K Gatford/UK/IBM@IBMGB
> To:
> "Amila Suriarachchi" <am...@gmail.com>
> Cc:
> "sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
> Date:
> 24/10/2008 11:07
> Subject:
> Re: Sandesha2 synchronization and dead lock handling.
>
>
>
> I went through similar pain when implementing a StorageManager and
> encountered a number of deadlocks similar to the ones that you describe.
> What I have gradually done is eliminate these in both the InMemory store
> and my store by changing the ordering the beans were taken in.
>
> In general the beans are taken in this order.
>
> RMSBean or RMDBean followed by
> SenderBean or InvokerBean.
>
> In cases where both the RMSBean and RMDBean are locked, they tend to be
> taken in that order - RMS followed by RMD.
> The one thing that I do know is that it is fairly easy to introduce new
> deadlocks by slightly altering the order that beans are read.
>
> The one question I have is how does the jdbc store handle multiple threads
>
> accessing multiple sequences, or even a single sequence, but with multiple
>
> threads sending multiple requests. From my experience this is where we
> have found a lot of problems in the InMemory store and I expect to be even
>
> more painful with a jdbc store.
>
> Andrew Gatford
> Technical Project Lead
> Websphere ESB Foundation Technologies
> Hursley MP211
> IBM United Kingdom Laboratories, Hursley Park, Winchester, SO21 2JN
> Telephone :
> Internal (7) 245743
> External 01962 815743
> Internet : gatfora@uk.ibm.com
>
>
>
> From:
> "Amila Suriarachchi" <am...@gmail.com>
> To:
> "sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
> Date:
> 24/10/2008 10:30
> Subject:
> Sandesha2 synchronization and dead lock handling.
>
>
>
> hi all,
>
> This is regarding the issue [1].
>
> First of all as I learned Sandesha2 uses different beans to keep the state
>
> of the sequence and the messages. In a dual channel mode
> different threads can access these beans and update them concurrently. So
> the synchronization of these beans done by using the
> storage level transactions. Therefore Sandesha2 needs an storage which
> supports isolated transactions.
>
> To synchronize these beans the transactions must be completely isolated.
> i.e It should not allow simultaneous reads of
> same record from different transactions. Therefore I think the problem I
> saw on[1] because not isolating the transactions properly.
>
> Then I increased the transaction isolation to fix the above problem. It
> fixed that problem but results in dead locks.
> The reason I believe for this dead locks is that different transactions
> try to access the data base tables in different order.
> But unfortunately I could not fix the issue.
>
> Normally these types of dead locks are prevented by accessing resources in
>
> same order. Does Sandesha2 follows such a order or any
> other technique?
>
> Or is there any other reason for this dead locks and synchronization
> problems? Can someone
> have a better idea of Sandesha2 Design shed some light on this?
>
> thanks,
> Amila.
>
>
> [1] http://issues.apache.org/jira/browse/SANDESHA2-179
> --
> Amila Suriarachchi
> WSO2 Inc.
> blog: http://amilachinthaka.blogspot.com/
>
>
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: sandesha-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: sandesha-dev-help@ws.apache.org
>
>
>
>
>
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
>
>
>
>
>
>
>
> --
> Amila Suriarachchi
> WSO2 Inc.
> blog: http://amilachinthaka.blogspot.com/
>
>
>
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
>
>
>
>
>
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Re: Sandesha2 synchronization and dead lock handling.
Posted by Thomas McKiernan <MC...@uk.ibm.com>.
I think Andrew was saying that there is already an implicit order for the
beans.
The point is people keep changing the code and breaking that order.
Therefore this is not quite a hack but an attempt to make explicit what is
already implied in the open src code.
It would not work for the distributed case but, assuming each distributed
thread was behaving correctly i.e. obeying the correct locking order then
this would not be an issue.
And this would catch any thread disobeying that ordering.
----------------------------------
Thomas McKiernan
WebSphere Messaging Development,
IBM United Kingdom Limited
Internal Phone: 248241
External Phone: +44 (0)1962 818241
Mobile: +44 (0)789 1737497
Email: MCKIERNA@uk.ibm.com
Mail Point 211, IBM, Hursley Park, Winchester, Hampshire, England, SO21
2JN
Caminante, no hay camino
Se hace camino al andar.
("Walker, there is no path; the path is made by walking.") Antonio
Machado
From:
"Amila Suriarachchi" <am...@gmail.com>
To:
Thomas McKiernan/UK/IBM@IBMGB
Cc:
Andrew K Gatford/UK/IBM@IBMGB, "sandesha-dev@ws.apache.org"
<sa...@ws.apache.org>
Date:
24/10/2008 14:06
Subject:
Re: Sandesha2 synchronization and dead lock handling.
On Fri, Oct 24, 2008 at 3:55 PM, Thomas McKiernan <MC...@uk.ibm.com>
wrote:
How about a lock manager impl independent of any particular store's impl.
It could be abstract if necessary.
Basically, this has a hierarchy of classes (beans) hard coded.
If you use a store to access a bean then the store impl's tran calls into
the independent lock manager.
I feel this is a kind hack for the problem. And also as Andrew has
mentioned this won't work in a
distributed environment.
For me the correct solution is to go through all the transactions and make
an order of which
transactions access the beans. But apparently this is also seems to be
difficult since a lot
of transactions has start and commits.
So have to think bit more.
thanks,
Amila.
Any attempt to enlist outside of the locking hierarchy results in a hard
runtime error and a rollback of the tran.
Is this too naive?
----------------------------------
Thomas McKiernan
WebSphere Messaging Development,
IBM United Kingdom Limited
Internal Phone: 248241
External Phone: +44 (0)1962 818241
Mobile: +44 (0)789 1737497
Email: MCKIERNA@uk.ibm.com
Mail Point 211, IBM, Hursley Park, Winchester, Hampshire, England, SO21
2JN
Caminante, no hay camino
Se hace camino al andar.
("Walker, there is no path; the path is made by walking.") Antonio
Machado
From:
Andrew K Gatford/UK/IBM@IBMGB
To:
"Amila Suriarachchi" <am...@gmail.com>
Cc:
"sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
Date:
24/10/2008 11:07
Subject:
Re: Sandesha2 synchronization and dead lock handling.
I went through similar pain when implementing a StorageManager and
encountered a number of deadlocks similar to the ones that you describe.
What I have gradually done is eliminate these in both the InMemory store
and my store by changing the ordering the beans were taken in.
In general the beans are taken in this order.
RMSBean or RMDBean followed by
SenderBean or InvokerBean.
In cases where both the RMSBean and RMDBean are locked, they tend to be
taken in that order - RMS followed by RMD.
The one thing that I do know is that it is fairly easy to introduce new
deadlocks by slightly altering the order that beans are read.
The one question I have is how does the jdbc store handle multiple threads
accessing multiple sequences, or even a single sequence, but with multiple
threads sending multiple requests. From my experience this is where we
have found a lot of problems in the InMemory store and I expect to be even
more painful with a jdbc store.
Andrew Gatford
Technical Project Lead
Websphere ESB Foundation Technologies
Hursley MP211
IBM United Kingdom Laboratories, Hursley Park, Winchester, SO21 2JN
Telephone :
Internal (7) 245743
External 01962 815743
Internet : gatfora@uk.ibm.com
From:
"Amila Suriarachchi" <am...@gmail.com>
To:
"sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
Date:
24/10/2008 10:30
Subject:
Sandesha2 synchronization and dead lock handling.
hi all,
This is regarding the issue [1].
First of all as I learned Sandesha2 uses different beans to keep the state
of the sequence and the messages. In a dual channel mode
different threads can access these beans and update them concurrently. So
the synchronization of these beans done by using the
storage level transactions. Therefore Sandesha2 needs an storage which
supports isolated transactions.
To synchronize these beans the transactions must be completely isolated.
i.e It should not allow simultaneous reads of
same record from different transactions. Therefore I think the problem I
saw on[1] because not isolating the transactions properly.
Then I increased the transaction isolation to fix the above problem. It
fixed that problem but results in dead locks.
The reason I believe for this dead locks is that different transactions
try to access the data base tables in different order.
But unfortunately I could not fix the issue.
Normally these types of dead locks are prevented by accessing resources in
same order. Does Sandesha2 follows such a order or any
other technique?
Or is there any other reason for this dead locks and synchronization
problems? Can someone
have a better idea of Sandesha2 Design shed some light on this?
thanks,
Amila.
[1] http://issues.apache.org/jira/browse/SANDESHA2-179
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
---------------------------------------------------------------------
To unsubscribe, e-mail: sandesha-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: sandesha-dev-help@ws.apache.org
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
---------------------------------------------------------------------
To unsubscribe, e-mail: sandesha-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: sandesha-dev-help@ws.apache.org
Re: Sandesha2 synchronization and dead lock handling.
Posted by Amila Suriarachchi <am...@gmail.com>.
On Fri, Oct 24, 2008 at 3:55 PM, Thomas McKiernan <MC...@uk.ibm.com>wrote:
> How about a lock manager impl independent of any particular store's impl.
> It could be abstract if necessary.
>
> Basically, this has a hierarchy of classes (beans) hard coded.
> If you use a store to access a bean then the store impl's tran calls into
> the independent lock manager.
I feel this is a kind hack for the problem. And also as Andrew has mentioned
this won't work in a
distributed environment.
For me the correct solution is to go through all the transactions and make
an order of which
transactions access the beans. But apparently this is also seems to be
difficult since a lot
of transactions has start and commits.
So have to think bit more.
thanks,
Amila.
>
> Any attempt to enlist outside of the locking hierarchy results in a hard
> runtime error and a rollback of the tran.
>
> Is this too naive?
>
> ----------------------------------
> Thomas McKiernan
>
> WebSphere Messaging Development,
> IBM United Kingdom Limited
>
> Internal Phone: 248241
> External Phone: +44 (0)1962 818241
> Mobile: +44 (0)789 1737497
> Email: MCKIERNA@uk.ibm.com
>
> Mail Point 211, IBM, Hursley Park, Winchester, Hampshire, England, SO21
> 2JN
>
>
> Caminante, no hay camino
> Se hace camino al andar.
> ("Walker, there is no path; the path is made by walking.") Antonio
> Machado
>
>
>
> From:
> Andrew K Gatford/UK/IBM@IBMGB
> To:
> "Amila Suriarachchi" <am...@gmail.com>
> Cc:
> "sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
> Date:
> 24/10/2008 11:07
> Subject:
> Re: Sandesha2 synchronization and dead lock handling.
>
>
>
> I went through similar pain when implementing a StorageManager and
> encountered a number of deadlocks similar to the ones that you describe.
> What I have gradually done is eliminate these in both the InMemory store
> and my store by changing the ordering the beans were taken in.
>
> In general the beans are taken in this order.
>
> RMSBean or RMDBean followed by
> SenderBean or InvokerBean.
>
> In cases where both the RMSBean and RMDBean are locked, they tend to be
> taken in that order - RMS followed by RMD.
> The one thing that I do know is that it is fairly easy to introduce new
> deadlocks by slightly altering the order that beans are read.
>
> The one question I have is how does the jdbc store handle multiple threads
>
> accessing multiple sequences, or even a single sequence, but with multiple
>
> threads sending multiple requests. From my experience this is where we
> have found a lot of problems in the InMemory store and I expect to be even
>
> more painful with a jdbc store.
>
> Andrew Gatford
> Technical Project Lead
> Websphere ESB Foundation Technologies
> Hursley MP211
> IBM United Kingdom Laboratories, Hursley Park, Winchester, SO21 2JN
> Telephone :
> Internal (7) 245743
> External 01962 815743
> Internet : gatfora@uk.ibm.com
>
>
>
> From:
> "Amila Suriarachchi" <am...@gmail.com>
> To:
> "sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
> Date:
> 24/10/2008 10:30
> Subject:
> Sandesha2 synchronization and dead lock handling.
>
>
>
> hi all,
>
> This is regarding the issue [1].
>
> First of all as I learned Sandesha2 uses different beans to keep the state
>
> of the sequence and the messages. In a dual channel mode
> different threads can access these beans and update them concurrently. So
> the synchronization of these beans done by using the
> storage level transactions. Therefore Sandesha2 needs an storage which
> supports isolated transactions.
>
> To synchronize these beans the transactions must be completely isolated.
> i.e It should not allow simultaneous reads of
> same record from different transactions. Therefore I think the problem I
> saw on[1] because not isolating the transactions properly.
>
> Then I increased the transaction isolation to fix the above problem. It
> fixed that problem but results in dead locks.
> The reason I believe for this dead locks is that different transactions
> try to access the data base tables in different order.
> But unfortunately I could not fix the issue.
>
> Normally these types of dead locks are prevented by accessing resources in
>
> same order. Does Sandesha2 follows such a order or any
> other technique?
>
> Or is there any other reason for this dead locks and synchronization
> problems? Can someone
> have a better idea of Sandesha2 Design shed some light on this?
>
> thanks,
> Amila.
>
>
> [1] http://issues.apache.org/jira/browse/SANDESHA2-179
> --
> Amila Suriarachchi
> WSO2 Inc.
> blog: http://amilachinthaka.blogspot.com/
>
>
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: sandesha-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: sandesha-dev-help@ws.apache.org
>
>
>
>
>
>
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
>
>
>
>
>
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Re: Sandesha2 synchronization and dead lock handling.
Posted by Thomas McKiernan <MC...@uk.ibm.com>.
How about a lock manager impl independent of any particular store's impl.
It could be abstract if necessary.
Basically, this has a hierarchy of classes (beans) hard coded.
If you use a store to access a bean then the store impl's tran calls into
the independent lock manager.
Any attempt to enlist outside of the locking hierarchy results in a hard
runtime error and a rollback of the tran.
Is this too naive?
----------------------------------
Thomas McKiernan
WebSphere Messaging Development,
IBM United Kingdom Limited
Internal Phone: 248241
External Phone: +44 (0)1962 818241
Mobile: +44 (0)789 1737497
Email: MCKIERNA@uk.ibm.com
Mail Point 211, IBM, Hursley Park, Winchester, Hampshire, England, SO21
2JN
Caminante, no hay camino
Se hace camino al andar.
("Walker, there is no path; the path is made by walking.") Antonio
Machado
From:
Andrew K Gatford/UK/IBM@IBMGB
To:
"Amila Suriarachchi" <am...@gmail.com>
Cc:
"sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
Date:
24/10/2008 11:07
Subject:
Re: Sandesha2 synchronization and dead lock handling.
I went through similar pain when implementing a StorageManager and
encountered a number of deadlocks similar to the ones that you describe.
What I have gradually done is eliminate these in both the InMemory store
and my store by changing the ordering the beans were taken in.
In general the beans are taken in this order.
RMSBean or RMDBean followed by
SenderBean or InvokerBean.
In cases where both the RMSBean and RMDBean are locked, they tend to be
taken in that order - RMS followed by RMD.
The one thing that I do know is that it is fairly easy to introduce new
deadlocks by slightly altering the order that beans are read.
The one question I have is how does the jdbc store handle multiple threads
accessing multiple sequences, or even a single sequence, but with multiple
threads sending multiple requests. From my experience this is where we
have found a lot of problems in the InMemory store and I expect to be even
more painful with a jdbc store.
Andrew Gatford
Technical Project Lead
Websphere ESB Foundation Technologies
Hursley MP211
IBM United Kingdom Laboratories, Hursley Park, Winchester, SO21 2JN
Telephone :
Internal (7) 245743
External 01962 815743
Internet : gatfora@uk.ibm.com
From:
"Amila Suriarachchi" <am...@gmail.com>
To:
"sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
Date:
24/10/2008 10:30
Subject:
Sandesha2 synchronization and dead lock handling.
hi all,
This is regarding the issue [1].
First of all as I learned Sandesha2 uses different beans to keep the state
of the sequence and the messages. In a dual channel mode
different threads can access these beans and update them concurrently. So
the synchronization of these beans done by using the
storage level transactions. Therefore Sandesha2 needs an storage which
supports isolated transactions.
To synchronize these beans the transactions must be completely isolated.
i.e It should not allow simultaneous reads of
same record from different transactions. Therefore I think the problem I
saw on[1] because not isolating the transactions properly.
Then I increased the transaction isolation to fix the above problem. It
fixed that problem but results in dead locks.
The reason I believe for this dead locks is that different transactions
try to access the data base tables in different order.
But unfortunately I could not fix the issue.
Normally these types of dead locks are prevented by accessing resources in
same order. Does Sandesha2 follows such a order or any
other technique?
Or is there any other reason for this dead locks and synchronization
problems? Can someone
have a better idea of Sandesha2 Design shed some light on this?
thanks,
Amila.
[1] http://issues.apache.org/jira/browse/SANDESHA2-179
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
---------------------------------------------------------------------
To unsubscribe, e-mail: sandesha-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: sandesha-dev-help@ws.apache.org
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
---------------------------------------------------------------------
To unsubscribe, e-mail: sandesha-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: sandesha-dev-help@ws.apache.org
Re: Sandesha2 synchronization and dead lock handling.
Posted by Andrew K Gatford <GA...@uk.ibm.com>.
I went through similar pain when implementing a StorageManager and
encountered a number of deadlocks similar to the ones that you describe.
What I have gradually done is eliminate these in both the InMemory store
and my store by changing the ordering the beans were taken in.
In general the beans are taken in this order.
RMSBean or RMDBean followed by
SenderBean or InvokerBean.
In cases where both the RMSBean and RMDBean are locked, they tend to be
taken in that order - RMS followed by RMD.
The one thing that I do know is that it is fairly easy to introduce new
deadlocks by slightly altering the order that beans are read.
The one question I have is how does the jdbc store handle multiple threads
accessing multiple sequences, or even a single sequence, but with multiple
threads sending multiple requests. From my experience this is where we
have found a lot of problems in the InMemory store and I expect to be even
more painful with a jdbc store.
Andrew Gatford
Technical Project Lead
Websphere ESB Foundation Technologies
Hursley MP211
IBM United Kingdom Laboratories, Hursley Park, Winchester, SO21 2JN
Telephone :
Internal (7) 245743
External 01962 815743
Internet : gatfora@uk.ibm.com
From:
"Amila Suriarachchi" <am...@gmail.com>
To:
"sandesha-dev@ws.apache.org" <sa...@ws.apache.org>
Date:
24/10/2008 10:30
Subject:
Sandesha2 synchronization and dead lock handling.
hi all,
This is regarding the issue [1].
First of all as I learned Sandesha2 uses different beans to keep the state
of the sequence and the messages. In a dual channel mode
different threads can access these beans and update them concurrently. So
the synchronization of these beans done by using the
storage level transactions. Therefore Sandesha2 needs an storage which
supports isolated transactions.
To synchronize these beans the transactions must be completely isolated.
i.e It should not allow simultaneous reads of
same record from different transactions. Therefore I think the problem I
saw on[1] because not isolating the transactions properly.
Then I increased the transaction isolation to fix the above problem. It
fixed that problem but results in dead locks.
The reason I believe for this dead locks is that different transactions
try to access the data base tables in different order.
But unfortunately I could not fix the issue.
Normally these types of dead locks are prevented by accessing resources in
same order. Does Sandesha2 follows such a order or any
other technique?
Or is there any other reason for this dead locks and synchronization
problems? Can someone
have a better idea of Sandesha2 Design shed some light on this?
thanks,
Amila.
[1] http://issues.apache.org/jira/browse/SANDESHA2-179
--
Amila Suriarachchi
WSO2 Inc.
blog: http://amilachinthaka.blogspot.com/
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
---------------------------------------------------------------------
To unsubscribe, e-mail: sandesha-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: sandesha-dev-help@ws.apache.org