You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by Imesh Gunaratne <im...@apache.org> on 2015/01/20 14:20:25 UTC
[Discuss] Detecting Issues with Read Write Locks
Hi Devs,
Currently we have significant amount of read write locks in Stratos
codebase to synchronize data structure updates made by multiple threads.
As I figured out recently there are several possibilities that these locks
could cause problems if not properly implemented:
*Problem 1: A thread trying to acquire a write lock while having a read
lock.*
*Problem 2: Locks acquired but not released*
I just implemented a ReadWriteLock class in stratos common module which
could detect these problems. This class make use
of java.util.concurrent.locks.ReentrantReadWriteLock and implements
additional logic to detect problems. By default problem 2 detection logic
is disabled, it can be enabled by setting following system property:
*read.write.lock.monitor.enabled=true*
I have now updated all read write locks with the above class.
Following is an example scenario where I detected an unreleased lock in the
REST API:
org.apache.stratos.common.exception.LockNotReleasedException
at
org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.checkTimeout(ReadWriteLockMonitor.java:71)
at
org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.run(ReadWriteLockMonitor.java:50)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2015-01-20 18:30:12,952] ERROR
{org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor} - System
error, lock has not released for 30 seconds: [lock-name] topology
[lock-type] Read [thread-id] 405 [thread-name] http-nio-9443-exec-21
[stack-trace]
java.lang.Thread.getStackTrace(Thread.java:1589)
org.apache.stratos.common.concurrent.locks.ReadWriteLock.acquireReadLock(ReadWriteLock.java:160)
org.apache.stratos.messaging.message.receiver.topology.TopologyManager.acquireReadLockForCluster(TopologyManager.java:173)
org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addClustersInstancesToGroupInstanceBean(StratosApiV41Utils.java:1015)
org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.setSubGroupInstances(StratosApiV41Utils.java:1027)
org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addGroupsInstancesToApplicationInstanceBean(StratosApiV41Utils.java:979)
org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.getApplicationRuntime(StratosApiV41Utils.java:963)
org.apache.stratos.rest.endpoint.api.StratosApiV41.getApplicationRuntime(StratosApiV41.java:579)
Thanks
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
Re: [Discuss] Detecting Issues with Read Write Locks
Posted by Imesh Gunaratne <im...@apache.org>.
It looks great! Thanks Mari!
On Wed, Jun 24, 2015 at 6:02 PM, Mariangela Hills <ma...@wso2.com>
wrote:
> Hi Devs,
>
> I added this information in [1] in the developer guide. Please let me know
> whether any changes are needed.
>
> [1]
> https://cwiki.apache.org/confluence/display/STRATOS/Detecting+Issues+with+Read-Write+Locks
> [2] https://issues.apache.org/jira/browse/STRATOS-1439
>
> Regards,
> Mariangela
>
>
> *Mariangela Hills*
> Senior Technical Writer - WSO2, Inc. http://wso2.com
> Committer and PMC member - Apache Stratos
> email:mariangela@wso2.com | mobile: +94 773 500185
>
> On Wed, Jun 24, 2015 at 3:10 PM, Mariangela Hills <ma...@wso2.com>
> wrote:
>
>> Will do!
>>
>> Regards,
>> Mariangela
>>
>>
>> *Mariangela Hills*
>> Senior Technical Writer - WSO2, Inc. http://wso2.com
>> Committer and PMC member - Apache Stratos
>> email:mariangela@wso2.com | mobile: +94 773 500185
>>
>> On Wed, Jun 24, 2015 at 2:43 PM, Reka Thirunavukkarasu <re...@wso2.com>
>> wrote:
>>
>>> Thanks for implementing this nice feature Imesh. This helps a lot in
>>> identifying the locking issues.
>>>
>>> @Mari,
>>>
>>> Can we add this to our documentation under developer guide or
>>> troubleshoot section? So that it will help in debugging purpose.
>>>
>>> Thanks,
>>> Reka
>>>
>>> On Tue, Jan 20, 2015 at 10:26 PM, Dakshika Jayathilaka <
>>> dakshika@wso2.com> wrote:
>>>
>>>> Great Work imesh.. This will help to developers in many ways..
>>>>
>>>> Better if we can add this to developer guide.
>>>>
>>>> *Dakshika Jayathilaka*
>>>> Software Engineer
>>>> WSO2, Inc.
>>>> lean.enterprise.middleware
>>>> 0771100911
>>>>
>>>> On Tue, Jan 20, 2015 at 7:08 PM, Imesh Gunaratne <im...@apache.org>
>>>> wrote:
>>>>
>>>>> Above issue is now fixed with commit revision:
>>>>> fe0e6f81a673d6db815a56a3592c41441c4be3ae
>>>>>
>>>>> On Tue, Jan 20, 2015 at 7:03 PM, Imesh Gunaratne <im...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Following JIRA explains the above problem found in 4.1.0-alpha:
>>>>>> https://issues.apache.org/jira/browse/STRATOS-1089
>>>>>>
>>>>>> On Tue, Jan 20, 2015 at 6:50 PM, Imesh Gunaratne <im...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Devs,
>>>>>>>
>>>>>>> Currently we have significant amount of read write locks in Stratos
>>>>>>> codebase to synchronize data structure updates made by multiple threads.
>>>>>>>
>>>>>>> As I figured out recently there are several possibilities that these
>>>>>>> locks could cause problems if not properly implemented:
>>>>>>>
>>>>>>> *Problem 1: A thread trying to acquire a write lock while having a
>>>>>>> read lock.*
>>>>>>> *Problem 2: Locks acquired but not released*
>>>>>>>
>>>>>>> I just implemented a ReadWriteLock class in stratos common module
>>>>>>> which could detect these problems. This class make use
>>>>>>> of java.util.concurrent.locks.ReentrantReadWriteLock and implements
>>>>>>> additional logic to detect problems. By default problem 2 detection logic
>>>>>>> is disabled, it can be enabled by setting following system property:
>>>>>>>
>>>>>>>
>>>>>>> *read.write.lock.monitor.enabled=true*
>>>>>>>
>>>>>>> I have now updated all read write locks with the above class.
>>>>>>> Following is an example scenario where I detected an unreleased lock
>>>>>>> in the REST API:
>>>>>>>
>>>>>>> org.apache.stratos.common.exception.LockNotReleasedException
>>>>>>> at
>>>>>>> org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.checkTimeout(ReadWriteLockMonitor.java:71)
>>>>>>> at
>>>>>>> org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.run(ReadWriteLockMonitor.java:50)
>>>>>>> at
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>> at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>> at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> [2015-01-20 18:30:12,952] ERROR
>>>>>>> {org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor} - System
>>>>>>> error, lock has not released for 30 seconds: [lock-name] topology
>>>>>>> [lock-type] Read [thread-id] 405 [thread-name] http-nio-9443-exec-21
>>>>>>> [stack-trace]
>>>>>>> java.lang.Thread.getStackTrace(Thread.java:1589)
>>>>>>>
>>>>>>> org.apache.stratos.common.concurrent.locks.ReadWriteLock.acquireReadLock(ReadWriteLock.java:160)
>>>>>>>
>>>>>>> org.apache.stratos.messaging.message.receiver.topology.TopologyManager.acquireReadLockForCluster(TopologyManager.java:173)
>>>>>>>
>>>>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addClustersInstancesToGroupInstanceBean(StratosApiV41Utils.java:1015)
>>>>>>>
>>>>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.setSubGroupInstances(StratosApiV41Utils.java:1027)
>>>>>>>
>>>>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addGroupsInstancesToApplicationInstanceBean(StratosApiV41Utils.java:979)
>>>>>>>
>>>>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.getApplicationRuntime(StratosApiV41Utils.java:963)
>>>>>>>
>>>>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41.getApplicationRuntime(StratosApiV41.java:579)
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Imesh Gunaratne
>>>>>>>
>>>>>>> Technical Lead, WSO2
>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Imesh Gunaratne
>>>>>>
>>>>>> Technical Lead, WSO2
>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Imesh Gunaratne
>>>>>
>>>>> Technical Lead, WSO2
>>>>> Committer & PMC Member, Apache Stratos
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Reka Thirunavukkarasu
>>> Senior Software Engineer,
>>> WSO2, Inc.:http://wso2.com,
>>> Mobile: +94776442007
>>>
>>>
>>>
>>
>
--
Imesh Gunaratne
Senior Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
Re: [Discuss] Detecting Issues with Read Write Locks
Posted by Mariangela Hills <ma...@wso2.com>.
Hi Devs,
I added this information in [1] in the developer guide. Please let me know
whether any changes are needed.
[1]
https://cwiki.apache.org/confluence/display/STRATOS/Detecting+Issues+with+Read-Write+Locks
[2] https://issues.apache.org/jira/browse/STRATOS-1439
Regards,
Mariangela
*Mariangela Hills*
Senior Technical Writer - WSO2, Inc. http://wso2.com
Committer and PMC member - Apache Stratos
email:mariangela@wso2.com | mobile: +94 773 500185
On Wed, Jun 24, 2015 at 3:10 PM, Mariangela Hills <ma...@wso2.com>
wrote:
> Will do!
>
> Regards,
> Mariangela
>
>
> *Mariangela Hills*
> Senior Technical Writer - WSO2, Inc. http://wso2.com
> Committer and PMC member - Apache Stratos
> email:mariangela@wso2.com | mobile: +94 773 500185
>
> On Wed, Jun 24, 2015 at 2:43 PM, Reka Thirunavukkarasu <re...@wso2.com>
> wrote:
>
>> Thanks for implementing this nice feature Imesh. This helps a lot in
>> identifying the locking issues.
>>
>> @Mari,
>>
>> Can we add this to our documentation under developer guide or
>> troubleshoot section? So that it will help in debugging purpose.
>>
>> Thanks,
>> Reka
>>
>> On Tue, Jan 20, 2015 at 10:26 PM, Dakshika Jayathilaka <dakshika@wso2.com
>> > wrote:
>>
>>> Great Work imesh.. This will help to developers in many ways..
>>>
>>> Better if we can add this to developer guide.
>>>
>>> *Dakshika Jayathilaka*
>>> Software Engineer
>>> WSO2, Inc.
>>> lean.enterprise.middleware
>>> 0771100911
>>>
>>> On Tue, Jan 20, 2015 at 7:08 PM, Imesh Gunaratne <im...@apache.org>
>>> wrote:
>>>
>>>> Above issue is now fixed with commit revision:
>>>> fe0e6f81a673d6db815a56a3592c41441c4be3ae
>>>>
>>>> On Tue, Jan 20, 2015 at 7:03 PM, Imesh Gunaratne <im...@apache.org>
>>>> wrote:
>>>>
>>>>> Following JIRA explains the above problem found in 4.1.0-alpha:
>>>>> https://issues.apache.org/jira/browse/STRATOS-1089
>>>>>
>>>>> On Tue, Jan 20, 2015 at 6:50 PM, Imesh Gunaratne <im...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi Devs,
>>>>>>
>>>>>> Currently we have significant amount of read write locks in Stratos
>>>>>> codebase to synchronize data structure updates made by multiple threads.
>>>>>>
>>>>>> As I figured out recently there are several possibilities that these
>>>>>> locks could cause problems if not properly implemented:
>>>>>>
>>>>>> *Problem 1: A thread trying to acquire a write lock while having a
>>>>>> read lock.*
>>>>>> *Problem 2: Locks acquired but not released*
>>>>>>
>>>>>> I just implemented a ReadWriteLock class in stratos common module
>>>>>> which could detect these problems. This class make use
>>>>>> of java.util.concurrent.locks.ReentrantReadWriteLock and implements
>>>>>> additional logic to detect problems. By default problem 2 detection logic
>>>>>> is disabled, it can be enabled by setting following system property:
>>>>>>
>>>>>>
>>>>>> *read.write.lock.monitor.enabled=true*
>>>>>>
>>>>>> I have now updated all read write locks with the above class.
>>>>>> Following is an example scenario where I detected an unreleased lock
>>>>>> in the REST API:
>>>>>>
>>>>>> org.apache.stratos.common.exception.LockNotReleasedException
>>>>>> at
>>>>>> org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.checkTimeout(ReadWriteLockMonitor.java:71)
>>>>>> at
>>>>>> org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.run(ReadWriteLockMonitor.java:50)
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>> at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> [2015-01-20 18:30:12,952] ERROR
>>>>>> {org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor} - System
>>>>>> error, lock has not released for 30 seconds: [lock-name] topology
>>>>>> [lock-type] Read [thread-id] 405 [thread-name] http-nio-9443-exec-21
>>>>>> [stack-trace]
>>>>>> java.lang.Thread.getStackTrace(Thread.java:1589)
>>>>>>
>>>>>> org.apache.stratos.common.concurrent.locks.ReadWriteLock.acquireReadLock(ReadWriteLock.java:160)
>>>>>>
>>>>>> org.apache.stratos.messaging.message.receiver.topology.TopologyManager.acquireReadLockForCluster(TopologyManager.java:173)
>>>>>>
>>>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addClustersInstancesToGroupInstanceBean(StratosApiV41Utils.java:1015)
>>>>>>
>>>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.setSubGroupInstances(StratosApiV41Utils.java:1027)
>>>>>>
>>>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addGroupsInstancesToApplicationInstanceBean(StratosApiV41Utils.java:979)
>>>>>>
>>>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.getApplicationRuntime(StratosApiV41Utils.java:963)
>>>>>>
>>>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41.getApplicationRuntime(StratosApiV41.java:579)
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Imesh Gunaratne
>>>>>>
>>>>>> Technical Lead, WSO2
>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Imesh Gunaratne
>>>>>
>>>>> Technical Lead, WSO2
>>>>> Committer & PMC Member, Apache Stratos
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Imesh Gunaratne
>>>>
>>>> Technical Lead, WSO2
>>>> Committer & PMC Member, Apache Stratos
>>>>
>>>
>>>
>>
>>
>> --
>> Reka Thirunavukkarasu
>> Senior Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>> Mobile: +94776442007
>>
>>
>>
>
Re: [Discuss] Detecting Issues with Read Write Locks
Posted by Mariangela Hills <ma...@wso2.com>.
Will do!
Regards,
Mariangela
*Mariangela Hills*
Senior Technical Writer - WSO2, Inc. http://wso2.com
Committer and PMC member - Apache Stratos
email:mariangela@wso2.com | mobile: +94 773 500185
On Wed, Jun 24, 2015 at 2:43 PM, Reka Thirunavukkarasu <re...@wso2.com>
wrote:
> Thanks for implementing this nice feature Imesh. This helps a lot in
> identifying the locking issues.
>
> @Mari,
>
> Can we add this to our documentation under developer guide or troubleshoot
> section? So that it will help in debugging purpose.
>
> Thanks,
> Reka
>
> On Tue, Jan 20, 2015 at 10:26 PM, Dakshika Jayathilaka <da...@wso2.com>
> wrote:
>
>> Great Work imesh.. This will help to developers in many ways..
>>
>> Better if we can add this to developer guide.
>>
>> *Dakshika Jayathilaka*
>> Software Engineer
>> WSO2, Inc.
>> lean.enterprise.middleware
>> 0771100911
>>
>> On Tue, Jan 20, 2015 at 7:08 PM, Imesh Gunaratne <im...@apache.org>
>> wrote:
>>
>>> Above issue is now fixed with commit revision:
>>> fe0e6f81a673d6db815a56a3592c41441c4be3ae
>>>
>>> On Tue, Jan 20, 2015 at 7:03 PM, Imesh Gunaratne <im...@apache.org>
>>> wrote:
>>>
>>>> Following JIRA explains the above problem found in 4.1.0-alpha:
>>>> https://issues.apache.org/jira/browse/STRATOS-1089
>>>>
>>>> On Tue, Jan 20, 2015 at 6:50 PM, Imesh Gunaratne <im...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Devs,
>>>>>
>>>>> Currently we have significant amount of read write locks in Stratos
>>>>> codebase to synchronize data structure updates made by multiple threads.
>>>>>
>>>>> As I figured out recently there are several possibilities that these
>>>>> locks could cause problems if not properly implemented:
>>>>>
>>>>> *Problem 1: A thread trying to acquire a write lock while having a
>>>>> read lock.*
>>>>> *Problem 2: Locks acquired but not released*
>>>>>
>>>>> I just implemented a ReadWriteLock class in stratos common module
>>>>> which could detect these problems. This class make use
>>>>> of java.util.concurrent.locks.ReentrantReadWriteLock and implements
>>>>> additional logic to detect problems. By default problem 2 detection logic
>>>>> is disabled, it can be enabled by setting following system property:
>>>>>
>>>>>
>>>>> *read.write.lock.monitor.enabled=true*
>>>>>
>>>>> I have now updated all read write locks with the above class.
>>>>> Following is an example scenario where I detected an unreleased lock
>>>>> in the REST API:
>>>>>
>>>>> org.apache.stratos.common.exception.LockNotReleasedException
>>>>> at
>>>>> org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.checkTimeout(ReadWriteLockMonitor.java:71)
>>>>> at
>>>>> org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.run(ReadWriteLockMonitor.java:50)
>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>> at
>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>> [2015-01-20 18:30:12,952] ERROR
>>>>> {org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor} - System
>>>>> error, lock has not released for 30 seconds: [lock-name] topology
>>>>> [lock-type] Read [thread-id] 405 [thread-name] http-nio-9443-exec-21
>>>>> [stack-trace]
>>>>> java.lang.Thread.getStackTrace(Thread.java:1589)
>>>>>
>>>>> org.apache.stratos.common.concurrent.locks.ReadWriteLock.acquireReadLock(ReadWriteLock.java:160)
>>>>>
>>>>> org.apache.stratos.messaging.message.receiver.topology.TopologyManager.acquireReadLockForCluster(TopologyManager.java:173)
>>>>>
>>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addClustersInstancesToGroupInstanceBean(StratosApiV41Utils.java:1015)
>>>>>
>>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.setSubGroupInstances(StratosApiV41Utils.java:1027)
>>>>>
>>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addGroupsInstancesToApplicationInstanceBean(StratosApiV41Utils.java:979)
>>>>>
>>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.getApplicationRuntime(StratosApiV41Utils.java:963)
>>>>>
>>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41.getApplicationRuntime(StratosApiV41.java:579)
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>> --
>>>>> Imesh Gunaratne
>>>>>
>>>>> Technical Lead, WSO2
>>>>> Committer & PMC Member, Apache Stratos
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Imesh Gunaratne
>>>>
>>>> Technical Lead, WSO2
>>>> Committer & PMC Member, Apache Stratos
>>>>
>>>
>>>
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>
>>
>
>
> --
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
> Mobile: +94776442007
>
>
>
Re: [Discuss] Detecting Issues with Read Write Locks
Posted by Reka Thirunavukkarasu <re...@wso2.com>.
Thanks for implementing this nice feature Imesh. This helps a lot in
identifying the locking issues.
@Mari,
Can we add this to our documentation under developer guide or troubleshoot
section? So that it will help in debugging purpose.
Thanks,
Reka
On Tue, Jan 20, 2015 at 10:26 PM, Dakshika Jayathilaka <da...@wso2.com>
wrote:
> Great Work imesh.. This will help to developers in many ways..
>
> Better if we can add this to developer guide.
>
> *Dakshika Jayathilaka*
> Software Engineer
> WSO2, Inc.
> lean.enterprise.middleware
> 0771100911
>
> On Tue, Jan 20, 2015 at 7:08 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> Above issue is now fixed with commit revision:
>> fe0e6f81a673d6db815a56a3592c41441c4be3ae
>>
>> On Tue, Jan 20, 2015 at 7:03 PM, Imesh Gunaratne <im...@apache.org>
>> wrote:
>>
>>> Following JIRA explains the above problem found in 4.1.0-alpha:
>>> https://issues.apache.org/jira/browse/STRATOS-1089
>>>
>>> On Tue, Jan 20, 2015 at 6:50 PM, Imesh Gunaratne <im...@apache.org>
>>> wrote:
>>>
>>>> Hi Devs,
>>>>
>>>> Currently we have significant amount of read write locks in Stratos
>>>> codebase to synchronize data structure updates made by multiple threads.
>>>>
>>>> As I figured out recently there are several possibilities that these
>>>> locks could cause problems if not properly implemented:
>>>>
>>>> *Problem 1: A thread trying to acquire a write lock while having a read
>>>> lock.*
>>>> *Problem 2: Locks acquired but not released*
>>>>
>>>> I just implemented a ReadWriteLock class in stratos common module which
>>>> could detect these problems. This class make use
>>>> of java.util.concurrent.locks.ReentrantReadWriteLock and implements
>>>> additional logic to detect problems. By default problem 2 detection logic
>>>> is disabled, it can be enabled by setting following system property:
>>>>
>>>>
>>>> *read.write.lock.monitor.enabled=true*
>>>>
>>>> I have now updated all read write locks with the above class.
>>>> Following is an example scenario where I detected an unreleased lock in
>>>> the REST API:
>>>>
>>>> org.apache.stratos.common.exception.LockNotReleasedException
>>>> at
>>>> org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.checkTimeout(ReadWriteLockMonitor.java:71)
>>>> at
>>>> org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.run(ReadWriteLockMonitor.java:50)
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> [2015-01-20 18:30:12,952] ERROR
>>>> {org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor} - System
>>>> error, lock has not released for 30 seconds: [lock-name] topology
>>>> [lock-type] Read [thread-id] 405 [thread-name] http-nio-9443-exec-21
>>>> [stack-trace]
>>>> java.lang.Thread.getStackTrace(Thread.java:1589)
>>>>
>>>> org.apache.stratos.common.concurrent.locks.ReadWriteLock.acquireReadLock(ReadWriteLock.java:160)
>>>>
>>>> org.apache.stratos.messaging.message.receiver.topology.TopologyManager.acquireReadLockForCluster(TopologyManager.java:173)
>>>>
>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addClustersInstancesToGroupInstanceBean(StratosApiV41Utils.java:1015)
>>>>
>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.setSubGroupInstances(StratosApiV41Utils.java:1027)
>>>>
>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addGroupsInstancesToApplicationInstanceBean(StratosApiV41Utils.java:979)
>>>>
>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.getApplicationRuntime(StratosApiV41Utils.java:963)
>>>>
>>>> org.apache.stratos.rest.endpoint.api.StratosApiV41.getApplicationRuntime(StratosApiV41.java:579)
>>>>
>>>> Thanks
>>>>
>>>>
>>>> --
>>>> Imesh Gunaratne
>>>>
>>>> Technical Lead, WSO2
>>>> Committer & PMC Member, Apache Stratos
>>>>
>>>
>>>
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
--
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007
Re: [Discuss] Detecting Issues with Read Write Locks
Posted by Dakshika Jayathilaka <da...@wso2.com>.
Great Work imesh.. This will help to developers in many ways..
Better if we can add this to developer guide.
*Dakshika Jayathilaka*
Software Engineer
WSO2, Inc.
lean.enterprise.middleware
0771100911
On Tue, Jan 20, 2015 at 7:08 PM, Imesh Gunaratne <im...@apache.org> wrote:
> Above issue is now fixed with commit revision:
> fe0e6f81a673d6db815a56a3592c41441c4be3ae
>
> On Tue, Jan 20, 2015 at 7:03 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> Following JIRA explains the above problem found in 4.1.0-alpha:
>> https://issues.apache.org/jira/browse/STRATOS-1089
>>
>> On Tue, Jan 20, 2015 at 6:50 PM, Imesh Gunaratne <im...@apache.org>
>> wrote:
>>
>>> Hi Devs,
>>>
>>> Currently we have significant amount of read write locks in Stratos
>>> codebase to synchronize data structure updates made by multiple threads.
>>>
>>> As I figured out recently there are several possibilities that these
>>> locks could cause problems if not properly implemented:
>>>
>>> *Problem 1: A thread trying to acquire a write lock while having a read
>>> lock.*
>>> *Problem 2: Locks acquired but not released*
>>>
>>> I just implemented a ReadWriteLock class in stratos common module which
>>> could detect these problems. This class make use
>>> of java.util.concurrent.locks.ReentrantReadWriteLock and implements
>>> additional logic to detect problems. By default problem 2 detection logic
>>> is disabled, it can be enabled by setting following system property:
>>>
>>>
>>> *read.write.lock.monitor.enabled=true*
>>>
>>> I have now updated all read write locks with the above class.
>>> Following is an example scenario where I detected an unreleased lock in
>>> the REST API:
>>>
>>> org.apache.stratos.common.exception.LockNotReleasedException
>>> at
>>> org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.checkTimeout(ReadWriteLockMonitor.java:71)
>>> at
>>> org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.run(ReadWriteLockMonitor.java:50)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>> [2015-01-20 18:30:12,952] ERROR
>>> {org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor} - System
>>> error, lock has not released for 30 seconds: [lock-name] topology
>>> [lock-type] Read [thread-id] 405 [thread-name] http-nio-9443-exec-21
>>> [stack-trace]
>>> java.lang.Thread.getStackTrace(Thread.java:1589)
>>>
>>> org.apache.stratos.common.concurrent.locks.ReadWriteLock.acquireReadLock(ReadWriteLock.java:160)
>>>
>>> org.apache.stratos.messaging.message.receiver.topology.TopologyManager.acquireReadLockForCluster(TopologyManager.java:173)
>>>
>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addClustersInstancesToGroupInstanceBean(StratosApiV41Utils.java:1015)
>>>
>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.setSubGroupInstances(StratosApiV41Utils.java:1027)
>>>
>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addGroupsInstancesToApplicationInstanceBean(StratosApiV41Utils.java:979)
>>>
>>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.getApplicationRuntime(StratosApiV41Utils.java:963)
>>>
>>> org.apache.stratos.rest.endpoint.api.StratosApiV41.getApplicationRuntime(StratosApiV41.java:579)
>>>
>>> Thanks
>>>
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
Re: [Discuss] Detecting Issues with Read Write Locks
Posted by Imesh Gunaratne <im...@apache.org>.
Above issue is now fixed with commit revision:
fe0e6f81a673d6db815a56a3592c41441c4be3ae
On Tue, Jan 20, 2015 at 7:03 PM, Imesh Gunaratne <im...@apache.org> wrote:
> Following JIRA explains the above problem found in 4.1.0-alpha:
> https://issues.apache.org/jira/browse/STRATOS-1089
>
> On Tue, Jan 20, 2015 at 6:50 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> Hi Devs,
>>
>> Currently we have significant amount of read write locks in Stratos
>> codebase to synchronize data structure updates made by multiple threads.
>>
>> As I figured out recently there are several possibilities that these
>> locks could cause problems if not properly implemented:
>>
>> *Problem 1: A thread trying to acquire a write lock while having a read
>> lock.*
>> *Problem 2: Locks acquired but not released*
>>
>> I just implemented a ReadWriteLock class in stratos common module which
>> could detect these problems. This class make use
>> of java.util.concurrent.locks.ReentrantReadWriteLock and implements
>> additional logic to detect problems. By default problem 2 detection logic
>> is disabled, it can be enabled by setting following system property:
>>
>>
>> *read.write.lock.monitor.enabled=true*
>>
>> I have now updated all read write locks with the above class.
>> Following is an example scenario where I detected an unreleased lock in
>> the REST API:
>>
>> org.apache.stratos.common.exception.LockNotReleasedException
>> at
>> org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.checkTimeout(ReadWriteLockMonitor.java:71)
>> at
>> org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.run(ReadWriteLockMonitor.java:50)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> [2015-01-20 18:30:12,952] ERROR
>> {org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor} - System
>> error, lock has not released for 30 seconds: [lock-name] topology
>> [lock-type] Read [thread-id] 405 [thread-name] http-nio-9443-exec-21
>> [stack-trace]
>> java.lang.Thread.getStackTrace(Thread.java:1589)
>>
>> org.apache.stratos.common.concurrent.locks.ReadWriteLock.acquireReadLock(ReadWriteLock.java:160)
>>
>> org.apache.stratos.messaging.message.receiver.topology.TopologyManager.acquireReadLockForCluster(TopologyManager.java:173)
>>
>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addClustersInstancesToGroupInstanceBean(StratosApiV41Utils.java:1015)
>>
>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.setSubGroupInstances(StratosApiV41Utils.java:1027)
>>
>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addGroupsInstancesToApplicationInstanceBean(StratosApiV41Utils.java:979)
>>
>> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.getApplicationRuntime(StratosApiV41Utils.java:963)
>>
>> org.apache.stratos.rest.endpoint.api.StratosApiV41.getApplicationRuntime(StratosApiV41.java:579)
>>
>> Thanks
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos
Re: [Discuss] Detecting Issues with Read Write Locks
Posted by Imesh Gunaratne <im...@apache.org>.
Following JIRA explains the above problem found in 4.1.0-alpha:
https://issues.apache.org/jira/browse/STRATOS-1089
On Tue, Jan 20, 2015 at 6:50 PM, Imesh Gunaratne <im...@apache.org> wrote:
> Hi Devs,
>
> Currently we have significant amount of read write locks in Stratos
> codebase to synchronize data structure updates made by multiple threads.
>
> As I figured out recently there are several possibilities that these locks
> could cause problems if not properly implemented:
>
> *Problem 1: A thread trying to acquire a write lock while having a read
> lock.*
> *Problem 2: Locks acquired but not released*
>
> I just implemented a ReadWriteLock class in stratos common module which
> could detect these problems. This class make use
> of java.util.concurrent.locks.ReentrantReadWriteLock and implements
> additional logic to detect problems. By default problem 2 detection logic
> is disabled, it can be enabled by setting following system property:
>
>
> *read.write.lock.monitor.enabled=true*
>
> I have now updated all read write locks with the above class.
> Following is an example scenario where I detected an unreleased lock in
> the REST API:
>
> org.apache.stratos.common.exception.LockNotReleasedException
> at
> org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.checkTimeout(ReadWriteLockMonitor.java:71)
> at
> org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.run(ReadWriteLockMonitor.java:50)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> [2015-01-20 18:30:12,952] ERROR
> {org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor} - System
> error, lock has not released for 30 seconds: [lock-name] topology
> [lock-type] Read [thread-id] 405 [thread-name] http-nio-9443-exec-21
> [stack-trace]
> java.lang.Thread.getStackTrace(Thread.java:1589)
>
> org.apache.stratos.common.concurrent.locks.ReadWriteLock.acquireReadLock(ReadWriteLock.java:160)
>
> org.apache.stratos.messaging.message.receiver.topology.TopologyManager.acquireReadLockForCluster(TopologyManager.java:173)
>
> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addClustersInstancesToGroupInstanceBean(StratosApiV41Utils.java:1015)
>
> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.setSubGroupInstances(StratosApiV41Utils.java:1027)
>
> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addGroupsInstancesToApplicationInstanceBean(StratosApiV41Utils.java:979)
>
> org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.getApplicationRuntime(StratosApiV41Utils.java:963)
>
> org.apache.stratos.rest.endpoint.api.StratosApiV41.getApplicationRuntime(StratosApiV41.java:579)
>
> Thanks
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PMC Member, Apache Stratos