You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@stratos.apache.org by Isuru Haththotuwa <is...@apache.org> on 2014/10/01 16:16:06 UTC

[Discuss] Hierarchical Locking for Topology

Hi Devs,

In the current Topology implementation, we acquire read/write locks on
Topology from the root level itself. For an example, if we need to modify a
single Cluster object, we still need to acquire a write lock from the
Topology root level. But, this is a costly operation. Specially, with
Service Grouping changes, we would need to traverse through an Application.
Since an Application can be a recursive structure, it can be a time
consuming operation. in such a scenario, if we are to lock the whole
Topology, there will be many threads waiting on that lock.

As a solution, I think we can use hierarchical locking [1]. For an example,
when you need to obtain the write lock for a particular Application, you do
not need to lock the whole tree, but can lock only that Application itself.
However, still we need to get the read locks for the parents.

A separate Lock tree will be maintained for the Topology.

Please share your feedback.

[1].
http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/2008/0801/071201hs01/071201hs01.html

-- 
Thanks and Regards,

Isuru H.
+94 716 358 048* <http://wso2.com/>*

Re: [Discuss] Hierarchical Locking for Topology

Posted by Imesh Gunaratne <im...@apache.org>.

Hi Isuru,

I think we need to be little careful when doing this, because the reason
for locking the complete tree at the root level was that we wanted to make
sure that operations that read the topology consider a snapshot of the tree
to take a decision.

If we manage locks in the middle of the tree, there is a possibility that
the topology may change while a process is traversing through it to execute
an operation.

Thanks

On Fri, Oct 3, 2014 at 3:05 PM, Isuru Haththotuwa <is...@apache.org> wrote:

> I did the initial changes, at the testing phase now. For an example, if we
> need to add a new Service, we do not need to lock everything now. We an
> only acquire the write lock on Services, and add the Service. If we need to
> modify a particular Service, can read lock the Services and acquire the
> write lock on the relevant Service and do the modification. This support is
> there for Services, Cluster and Applications.
>
> On Wed, Oct 1, 2014 at 9:27 PM, Manula Chathurika Thantriwatte <
> manulac@wso2.com> wrote:
>
>> Hi Isuru,
>>
>> +1 for the hierarchical locking approach. Using hierarchical locking we
>> can have more benefits [1].
>>
>> [1]
>> http://synapticnulship.com/blog/2013/03/08/comparison-chainlocker-vs-heirarchical-mutexes/
>>
>>
>> On Wed, Oct 1, 2014 at 7:46 PM, Isuru Haththotuwa <is...@apache.org>
>> wrote:
>>
>>> Hi Devs,
>>>
>>> In the current Topology implementation, we acquire read/write locks on
>>> Topology from the root level itself. For an example, if we need to modify a
>>> single Cluster object, we still need to acquire a write lock from the
>>> Topology root level. But, this is a costly operation. Specially, with
>>> Service Grouping changes, we would need to traverse through an Application.
>>> Since an Application can be a recursive structure, it can be a time
>>> consuming operation. in such a scenario, if we are to lock the whole
>>> Topology, there will be many threads waiting on that lock.
>>>
>>> As a solution, I think we can use hierarchical locking [1]. For an
>>> example, when you need to obtain the write lock for a particular
>>> Application, you do not need to lock the whole tree, but can lock only that
>>> Application itself. However, still we need to get the read locks for the
>>> parents.
>>>
>>> A separate Lock tree will be maintained for the Topology.
>>>
>>> Please share your feedback.
>>>
>>> [1].
>>> http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/2008/0801/071201hs01/071201hs01.html
>>>
>>> --
>>> Thanks and Regards,
>>>
>>> Isuru H.
>>> +94 716 358 048* <http://wso2.com/>*
>>>
>>>
>>>
>>
>>
>> --
>> Regards,
>> Manula Chathurika Thantriwatte
>> Software Engineer
>> WSO2 Inc. : http://wso2.com
>> lean . enterprise . middleware
>>
>> email : manulac@wso2.com / manula@apache.org
>> phone : +94 772492511
>> blog : http://manulachathurika.blogspot.com/
>>
>> --
>> <http://manulachathurika.blogspot.com/>
>> Thanks and Regards,
>>
>> Isuru H.
>> <http://manulachathurika.blogspot.com/>
>> +94 716 358 048 <http://manulachathurika.blogspot.com/>*
>> <http://wso2.com/>*
>>
>>
>> * <http://wso2.com/>*
>>
>>
>>


-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

Re: [Discuss] Hierarchical Locking for Topology

Posted by Isuru Haththotuwa <is...@apache.org>.

Thanks Gayan.

One another thing I see is that in the TopologyEventMessageDelegator, we
use a single thread to get messages from the queue and run them through the
TopologyMessageProcessorChain. This again would limit the ability to to
parallel updates to the Topology IMHO. To solve this, we can use a thread
pool to retrieve messages from the queue and run them through the
TopologyMessageProcessorChain, WDYT?

On Tue, Oct 14, 2014 at 4:34 PM, Gayan Gunarathne <ga...@wso2.com> wrote:

> Hi Isuru,
>
> I mean that if you try to add the hierarchical locking with the topology
> update, it is better if we can have idea about how many siblings
> independently we can lock with updating topology.
> But if CC allows only the sequential access IMO we can't get advantage
> from the hierarchical locking.
>
> Thanks,
> Gayan
>
>
>
>
> On Tue, Oct 14, 2014 at 3:47 PM, Isuru Haththotuwa <is...@apache.org>
> wrote:
>
>> Hi Gayan,
>>
>> On Mon, Oct 13, 2014 at 7:50 PM, Gayan Gunarathne <ga...@wso2.com>
>> wrote:
>>
>>> IMO it is based on how many siblings we have in the topology which we
>>> can lock independently. If we have lot of occurrences where we can able
>>> to lock the siblings independently it will improve the performance.But
>>> again I think we need to careful with the dead lock scenarios.
>>>
>> I did not understand your statement exactly. Can you please explain a bit
>> more?
>>
>> My understanding is, if CC allows only sequential access to its own
>> Topology model, then that will again be a bottleneck since CC is the one
>> who sends out the Topology events.
>>
>>>
>>> Thanks,
>>> Gayan
>>>
>>> On Mon, Oct 13, 2014 at 5:50 PM, Isuru Haththotuwa <is...@apache.org>
>>> wrote:
>>>
>>>> Another improvement that we can do is to introduce the same
>>>> hierarchical locking to the Topology structure maintained in the Cloud
>>>> Controller. Its actually the CC who will be updating the local Topology and
>>>> sending the relevant events. In that case, if we have have only one write
>>>> lock for the whole Topology in the CC's TopologyBuilder, it will still be a
>>>> bottleneck. WDYT?
>>>>
>>>> On Thu, Oct 9, 2014 at 3:36 PM, Isuru Haththotuwa <is...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>> Since the Topology is updated from the messaging component only, I
>>>>> removed the methods to access the write locks to an internal class. Those
>>>>> methods will not be exposed to outside by the messaging component. Now, the
>>>>> following methods to obtain read-only locks are exposed from
>>>>> TopologyManager:
>>>>>
>>>>>     // Topology level read locks
>>>>>
>>>>>     /**
>>>>>      * Acquires read lock for the Complete Topology
>>>>>      */
>>>>> *    public static void acquireReadLock() ;*
>>>>>
>>>>>     /**
>>>>>      * Releases read lock for the Complete Topology
>>>>>      */
>>>>> *    public static void releaseReadLock();*
>>>>>
>>>>>     // Application and Service read locks
>>>>>
>>>>>     /**
>>>>>      * Acquires read lock for the all Applications
>>>>>      */
>>>>> *    public static void acquireReadLockForApplications() ;*
>>>>>
>>>>>     /**
>>>>>      * Releases read lock for the all Applications
>>>>>      */
>>>>> *    public static void releaseReadLockForApplications();*
>>>>>     /**
>>>>>      * Acquires read lock for the all Services
>>>>>      */
>>>>> *    public static void acquireReadLockForServices();*
>>>>>
>>>>>     /**
>>>>>      * Releases read lock for the all Services
>>>>>      */
>>>>> *    public static void releaseReadLockForServices() ;*
>>>>>
>>>>>     /**
>>>>>      * Acquires read lock for a Service
>>>>>      *
>>>>>      * @param serviceName service name to acquire read lock
>>>>>      */
>>>>> *    public static void acquireReadLockForService (String serviceName)
>>>>> ;*
>>>>>
>>>>>     /**
>>>>>      * Releases read lock for a Service
>>>>>      *
>>>>>      * @param serviceName service name to release read lock
>>>>>      */
>>>>> *    public static void releaseReadLockForService (String serviceName)
>>>>> ;*
>>>>>
>>>>>     /**
>>>>>      * Acquires read lock for a Cluster. This will acquire the read
>>>>> lock in the following order
>>>>>      *      1. for the Service
>>>>>      *      2. for the Cluster
>>>>>      *
>>>>>      * @param serviceName service name to acquire read lock
>>>>>      * @param clusterId cluster id to acquire read lock
>>>>>      */
>>>>> *    public static void acquireReadLockForCluster (String serviceName,
>>>>> String clusterId);*
>>>>>
>>>>>     /**
>>>>>      * Releases read lock for a Cluster. This will release the read
>>>>> lock in the following order
>>>>>      *      1. for the Cluster
>>>>>      *      2. for the Service
>>>>>      *
>>>>>      * @param serviceName service name to release read lock
>>>>>      * @param clusterId cluster id to release read lock
>>>>>      */
>>>>> *    public static void releaseReadLockForCluster (String serviceName,
>>>>> String clusterId);*
>>>>>
>>>>>     /**
>>>>>      * Acquires read lock for the Application
>>>>>      *
>>>>>      * @param appId Application id
>>>>>      */
>>>>> *    public static void acquireReadLockForApplication (String appId) ;*
>>>>>
>>>>>     /**
>>>>>      * Releases read lock for the Application
>>>>>      *
>>>>>      * @param appId Application id
>>>>>      */
>>>>> *    public static void releaseReadLockForApplication (String appId);*
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 6, 2014 at 2:51 PM, Imesh Gunaratne <im...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> This looks great! As we discussed may be we could provide an
>>>>>> interface in the messaging component to acquire and release locks at
>>>>>> different sub tree levels. The whole idea is to avoid any possibilities of
>>>>>> reading the topology in an inconsistent state.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Mon, Oct 6, 2014 at 12:37 PM, Isuru Haththotuwa <isuruh@apache.org
>>>>>> > wrote:
>>>>>>
>>>>>>> Thanks Imesh.
>>>>>>>
>>>>>>> I have shown some examples for the new hierarchical locking
>>>>>>> approach. Please do let me know your feedback on this.
>>>>>>>
>>>>>>>    Acquire a write lock:
>>>>>>>
>>>>>>>    From root level, acquire read lock, and acquire a write lock only
>>>>>>> for the
>>>>>>>    relevant sub tree.
>>>>>>>
>>>>>>>    Acquire a read lock:
>>>>>>>
>>>>>>>    From root level, acquire read locks till the relevant sub tree
>>>>>>>
>>>>>>>    Examples -
>>>>>>>
>>>>>>>    Example 1: Acquiring write lock for a Cluster to modify the
>>>>>>> Cluster object -
>>>>>>>             acquiring:
>>>>>>>             1. acquire read lock for all Services
>>>>>>>             2. acquire read lock for the particular Service, to
>>>>>>> which the cluster belongs to
>>>>>>>             3. acquire write lock for the Cluster
>>>>>>>
>>>>>>>             releasing:
>>>>>>>             1. release write lock for the Cluster
>>>>>>>             2. release read lock for the particular Service
>>>>>>>             3. release read lock for all Services
>>>>>>>
>>>>>>>    Example 2: Acquiring write lock to add a new Cluster object -
>>>>>>>             acquiring:
>>>>>>>             1. acquire read lock for all Services
>>>>>>>             2. acquire write lock for the particular Service, to
>>>>>>> which the cluster belongs to
>>>>>>>
>>>>>>>             releasing:
>>>>>>>             1. release write lock for the particular Service
>>>>>>>             2. release read lock for all Services
>>>>>>>
>>>>>>>    Example 3: Acquiring read lock to read Cluster information
>>>>>>>             acquiring:
>>>>>>>             1. acquire read lock for all Services
>>>>>>>             2. acquire read lock for the particular Service, to
>>>>>>> which the cluster belongs to
>>>>>>>             3. acquire read lock for the relevant Cluster
>>>>>>>
>>>>>>>             releasing:
>>>>>>>             1. release read lock for the relevant Cluster
>>>>>>>             2. release read lock for the particular Service
>>>>>>>             3. release read lock for all Services
>>>>>>>
>>>>>>>    Example 4: Acquiring the write lock to add a deploy a Cartridge
>>>>>>> (add a new Service)
>>>>>>>             acquire:
>>>>>>>             1. acquire write lock for all Services
>>>>>>>
>>>>>>>             release:
>>>>>>>             1. release write lock for all Services
>>>>>>>
>>>>>>> In all of these examples, the lock acquiring happens from top of the
>>>>>>> tree to down. This is to avoid deadlocks.
>>>>>>>
>>>>>>> On Mon, Oct 6, 2014 at 10:50 AM, Imesh Gunaratne <im...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Great! Thanks for the clarification Isuru!
>>>>>>>>
>>>>>>>> Yes I agree, I think what we can do is, identify the sub trees that
>>>>>>>> will not break the consistency of the data structure and manage locks at
>>>>>>>> those sub tree level.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> On Sun, Oct 5, 2014 at 8:07 PM, Isuru Haththotuwa <
>>>>>>>> isuruh@apache.org> wrote:
>>>>>>>>
>>>>>>>>> Hi Lahiru and Imesh,
>>>>>>>>>
>>>>>>>>> Thanks a lot for the input.
>>>>>>>>>
>>>>>>>>> What I do here is locking only the relevant sub tree of the
>>>>>>>>> complete Topology tree, as locking the whole tree is rather inefficient.
>>>>>>>>> For an example, when a MemberActivated event is received, we have the
>>>>>>>>> cluster id of the cluster that particular member belongs to. IMHO, we only
>>>>>>>>> need to acquire the write lock for that cluster , and do not need the lock
>>>>>>>>> for complete Topology tree. Therefore, any other thread which needs to do
>>>>>>>>> another operation on a separate sub tree (for an example, deploy a new
>>>>>>>>> service, etc.) can do that concurrently.
>>>>>>>>>
>>>>>>>>> On Sun, Oct 5, 2014 at 12:20 AM, Lahiru Sandaruwan <
>>>>>>>>> lahirus@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Isuru,
>>>>>>>>>>
>>>>>>>>>> Looks like a good move to improve the efficiency,
>>>>>>>>>>
>>>>>>>>>> On Fri, Oct 3, 2014 at 3:05 PM, Isuru Haththotuwa <
>>>>>>>>>> isuruh@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> I did the initial changes, at the testing phase now. For an
>>>>>>>>>>> example, if we need to add a new Service, we do not need to lock everything
>>>>>>>>>>> now. We an only acquire the write lock on Services, and add the Service.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Service can be an element of a group or an app. So shouldn't we
>>>>>>>>>> lock complete application if we add/modify a service? Otherwise a different
>>>>>>>>>> thread might change parents/relationships otherwise.
>>>>>>>>>>
>>>>>>>>> AFAIK a Service gets created when we deploy a cartridge. An
>>>>>>>>> application/Group can refer a service. In the case of modifying an
>>>>>>>>> Application, we do need to lock the relevant clusters that belong to that
>>>>>>>>> Application. I implemented that.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> So generally i think we can bring down the locking level to
>>>>>>>>>> Application, but not the services. Also if we need to read any part, we
>>>>>>>>>> have to get the read lock for the whole topology, such that the receiver
>>>>>>>>>> get a particular snapshot of the topology as Imesh also mentioned.
>>>>>>>>>>
>>>>>>>>> If we need to lock the complete Topology, we can still do that,
>>>>>>>>> such as in a Complete Topology event. But IMHO, if we know the exact part
>>>>>>>>> (a particular Cluster, etc.) that we need to read/write, we do not need to
>>>>>>>>> lock the whole Topology.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> If we need to modify a particular Service, can read lock the
>>>>>>>>>>> Services and acquire the write lock on the relevant Service and do the
>>>>>>>>>>> modification.
>>>>>>>>>>>
>>>>>>>>>> This support is there for Services, Cluster and Applications.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On Wed, Oct 1, 2014 at 9:27 PM, Manula Chathurika Thantriwatte <
>>>>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Isuru,
>>>>>>>>>>>>
>>>>>>>>>>>> +1 for the hierarchical locking approach. Using hierarchical
>>>>>>>>>>>> locking we can have more benefits [1].
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> http://synapticnulship.com/blog/2013/03/08/comparison-chainlocker-vs-heirarchical-mutexes/
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Oct 1, 2014 at 7:46 PM, Isuru Haththotuwa <
>>>>>>>>>>>> isuruh@apache.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Devs,
>>>>>>>>>>>>>
>>>>>>>>>>>>> In the current Topology implementation, we acquire read/write
>>>>>>>>>>>>> locks on Topology from the root level itself. For an example, if we need to
>>>>>>>>>>>>> modify a single Cluster object, we still need to acquire a write lock from
>>>>>>>>>>>>> the Topology root level. But, this is a costly operation. Specially, with
>>>>>>>>>>>>> Service Grouping changes, we would need to traverse through an Application.
>>>>>>>>>>>>> Since an Application can be a recursive structure, it can be a time
>>>>>>>>>>>>> consuming operation. in such a scenario, if we are to lock the whole
>>>>>>>>>>>>> Topology, there will be many threads waiting on that lock.
>>>>>>>>>>>>>
>>>>>>>>>>>>> As a solution, I think we can use hierarchical locking [1].
>>>>>>>>>>>>> For an example, when you need to obtain the write lock for a particular
>>>>>>>>>>>>> Application, you do not need to lock the whole tree, but can lock only that
>>>>>>>>>>>>> Application itself. However, still we need to get the read locks for the
>>>>>>>>>>>>> parents.
>>>>>>>>>>>>>
>>>>>>>>>>>>> A separate Lock tree will be maintained for the Topology.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please share your feedback.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1].
>>>>>>>>>>>>> http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/2008/0801/071201hs01/071201hs01.html
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Isuru H.
>>>>>>>>>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>>> Software Engineer
>>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>>
>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> <http://manulachathurika.blogspot.com/>
>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Isuru H.
>>>>>>>>>>>> <http://manulachathurika.blogspot.com/>
>>>>>>>>>>>> +94 716 358 048 <http://manulachathurika.blogspot.com/>*
>>>>>>>>>>>> <http://wso2.com/>*
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> * <http://wso2.com/>*
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> --
>>>>>>>>>> Lahiru Sandaruwan
>>>>>>>>>> Committer and PMC member, Apache Stratos,
>>>>>>>>>> Senior Software Engineer,
>>>>>>>>>> WSO2 Inc., http://wso2.com
>>>>>>>>>> lean.enterprise.middleware
>>>>>>>>>>
>>>>>>>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>>>>>>>> blog: http://lahiruwrites.blogspot.com/
>>>>>>>>>> twitter: http://twitter.com/lahirus
>>>>>>>>>> linked-in:
>>>>>>>>>> http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>
>>>>>>>>>> Isuru H.
>>>>>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>>>>>>>>>> +94 716 358 048
>>>>>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>*
>>>>>>>>>> <http://wso2.com/>*
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> * <http://wso2.com/>*
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Imesh Gunaratne
>>>>>>>>
>>>>>>>> Technical Lead, WSO2
>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks and Regards,
>>>>>>>>
>>>>>>>> Isuru H.
>>>>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>>>>
>>>>>>>>
>>>>>>>> * <http://wso2.com/>*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Imesh Gunaratne
>>>>>>
>>>>>> Technical Lead, WSO2
>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>
>>>>>> --
>>>>>> Thanks and Regards,
>>>>>>
>>>>>> Isuru H.
>>>>>> +94 716 358 048
>>>>>>
>>>>>> --
>>>>>> <%2B94%20716%20358%20048>
>>>>>> Thanks and Regards,
>>>>>>
>>>>>> Isuru H.
>>>>>> <%2B94%20716%20358%20048>
>>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>>
>>>>>>
>>>>>> * <http://wso2.com/>*
>>>>>>
>>>>>>
>>>>>>
>>>
>>>
>>> --
>>>
>>> Gayan Gunarathne
>>> Technical Lead
>>> WSO2 Inc. (http://wso2.com)
>>> email  : gayang@wso2.com  | mobile : +94 766819985
>>>
>>> --
>>> <%2B94%20766819985>
>>> Thanks and Regards,
>>>
>>> Isuru H.
>>> <%2B94%20766819985>
>>> +94 716 358 048 <%2B94%20766819985>* <http://wso2.com/>*
>>>
>>>
>>> * <http://wso2.com/>*
>>>
>>>
>>>
>
>
> --
>
> Gayan Gunarathne
> Technical Lead
> WSO2 Inc. (http://wso2.com)
> email  : gayang@wso2.com  | mobile : +94 766819985
>
> --
> Thanks and Regards,
>
> Isuru H.
> +94 716 358 048* <http://wso2.com/>*
>
>
> * <http://wso2.com/>*
>
>
>

Re: [Discuss] Hierarchical Locking for Topology

Posted by Gayan Gunarathne <ga...@wso2.com>.

Hi Isuru,

I mean that if you try to add the hierarchical locking with the topology
update, it is better if we can have idea about how many siblings
independently we can lock with updating topology.
But if CC allows only the sequential access IMO we can't get advantage from
the hierarchical locking.

Thanks,
Gayan




On Tue, Oct 14, 2014 at 3:47 PM, Isuru Haththotuwa <is...@apache.org>
wrote:

> Hi Gayan,
>
> On Mon, Oct 13, 2014 at 7:50 PM, Gayan Gunarathne <ga...@wso2.com> wrote:
>
>> IMO it is based on how many siblings we have in the topology which we can
>> lock independently. If we have lot of occurrences where we can able to
>> lock the siblings independently it will improve the performance.But again I
>> think we need to careful with the dead lock scenarios.
>>
> I did not understand your statement exactly. Can you please explain a bit
> more?
>
> My understanding is, if CC allows only sequential access to its own
> Topology model, then that will again be a bottleneck since CC is the one
> who sends out the Topology events.
>
>>
>> Thanks,
>> Gayan
>>
>> On Mon, Oct 13, 2014 at 5:50 PM, Isuru Haththotuwa <is...@apache.org>
>> wrote:
>>
>>> Another improvement that we can do is to introduce the same hierarchical
>>> locking to the Topology structure maintained in the Cloud Controller. Its
>>> actually the CC who will be updating the local Topology and sending the
>>> relevant events. In that case, if we have have only one write lock for the
>>> whole Topology in the CC's TopologyBuilder, it will still be a bottleneck.
>>> WDYT?
>>>
>>> On Thu, Oct 9, 2014 at 3:36 PM, Isuru Haththotuwa <is...@apache.org>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>>
>>>> Since the Topology is updated from the messaging component only, I
>>>> removed the methods to access the write locks to an internal class. Those
>>>> methods will not be exposed to outside by the messaging component. Now, the
>>>> following methods to obtain read-only locks are exposed from
>>>> TopologyManager:
>>>>
>>>>     // Topology level read locks
>>>>
>>>>     /**
>>>>      * Acquires read lock for the Complete Topology
>>>>      */
>>>> *    public static void acquireReadLock() ;*
>>>>
>>>>     /**
>>>>      * Releases read lock for the Complete Topology
>>>>      */
>>>> *    public static void releaseReadLock();*
>>>>
>>>>     // Application and Service read locks
>>>>
>>>>     /**
>>>>      * Acquires read lock for the all Applications
>>>>      */
>>>> *    public static void acquireReadLockForApplications() ;*
>>>>
>>>>     /**
>>>>      * Releases read lock for the all Applications
>>>>      */
>>>> *    public static void releaseReadLockForApplications();*
>>>>     /**
>>>>      * Acquires read lock for the all Services
>>>>      */
>>>> *    public static void acquireReadLockForServices();*
>>>>
>>>>     /**
>>>>      * Releases read lock for the all Services
>>>>      */
>>>> *    public static void releaseReadLockForServices() ;*
>>>>
>>>>     /**
>>>>      * Acquires read lock for a Service
>>>>      *
>>>>      * @param serviceName service name to acquire read lock
>>>>      */
>>>> *    public static void acquireReadLockForService (String serviceName)
>>>> ;*
>>>>
>>>>     /**
>>>>      * Releases read lock for a Service
>>>>      *
>>>>      * @param serviceName service name to release read lock
>>>>      */
>>>> *    public static void releaseReadLockForService (String serviceName)
>>>> ;*
>>>>
>>>>     /**
>>>>      * Acquires read lock for a Cluster. This will acquire the read
>>>> lock in the following order
>>>>      *      1. for the Service
>>>>      *      2. for the Cluster
>>>>      *
>>>>      * @param serviceName service name to acquire read lock
>>>>      * @param clusterId cluster id to acquire read lock
>>>>      */
>>>> *    public static void acquireReadLockForCluster (String serviceName,
>>>> String clusterId);*
>>>>
>>>>     /**
>>>>      * Releases read lock for a Cluster. This will release the read
>>>> lock in the following order
>>>>      *      1. for the Cluster
>>>>      *      2. for the Service
>>>>      *
>>>>      * @param serviceName service name to release read lock
>>>>      * @param clusterId cluster id to release read lock
>>>>      */
>>>> *    public static void releaseReadLockForCluster (String serviceName,
>>>> String clusterId);*
>>>>
>>>>     /**
>>>>      * Acquires read lock for the Application
>>>>      *
>>>>      * @param appId Application id
>>>>      */
>>>> *    public static void acquireReadLockForApplication (String appId) ;*
>>>>
>>>>     /**
>>>>      * Releases read lock for the Application
>>>>      *
>>>>      * @param appId Application id
>>>>      */
>>>> *    public static void releaseReadLockForApplication (String appId);*
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Oct 6, 2014 at 2:51 PM, Imesh Gunaratne <im...@apache.org>
>>>> wrote:
>>>>
>>>>> This looks great! As we discussed may be we could provide an interface
>>>>> in the messaging component to acquire and release locks at different sub
>>>>> tree levels. The whole idea is to avoid any possibilities of reading the
>>>>> topology in an inconsistent state.
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Mon, Oct 6, 2014 at 12:37 PM, Isuru Haththotuwa <is...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Thanks Imesh.
>>>>>>
>>>>>> I have shown some examples for the new hierarchical locking approach.
>>>>>> Please do let me know your feedback on this.
>>>>>>
>>>>>>    Acquire a write lock:
>>>>>>
>>>>>>    From root level, acquire read lock, and acquire a write lock only
>>>>>> for the
>>>>>>    relevant sub tree.
>>>>>>
>>>>>>    Acquire a read lock:
>>>>>>
>>>>>>    From root level, acquire read locks till the relevant sub tree
>>>>>>
>>>>>>    Examples -
>>>>>>
>>>>>>    Example 1: Acquiring write lock for a Cluster to modify the
>>>>>> Cluster object -
>>>>>>             acquiring:
>>>>>>             1. acquire read lock for all Services
>>>>>>             2. acquire read lock for the particular Service, to which
>>>>>> the cluster belongs to
>>>>>>             3. acquire write lock for the Cluster
>>>>>>
>>>>>>             releasing:
>>>>>>             1. release write lock for the Cluster
>>>>>>             2. release read lock for the particular Service
>>>>>>             3. release read lock for all Services
>>>>>>
>>>>>>    Example 2: Acquiring write lock to add a new Cluster object -
>>>>>>             acquiring:
>>>>>>             1. acquire read lock for all Services
>>>>>>             2. acquire write lock for the particular Service, to
>>>>>> which the cluster belongs to
>>>>>>
>>>>>>             releasing:
>>>>>>             1. release write lock for the particular Service
>>>>>>             2. release read lock for all Services
>>>>>>
>>>>>>    Example 3: Acquiring read lock to read Cluster information
>>>>>>             acquiring:
>>>>>>             1. acquire read lock for all Services
>>>>>>             2. acquire read lock for the particular Service, to which
>>>>>> the cluster belongs to
>>>>>>             3. acquire read lock for the relevant Cluster
>>>>>>
>>>>>>             releasing:
>>>>>>             1. release read lock for the relevant Cluster
>>>>>>             2. release read lock for the particular Service
>>>>>>             3. release read lock for all Services
>>>>>>
>>>>>>    Example 4: Acquiring the write lock to add a deploy a Cartridge
>>>>>> (add a new Service)
>>>>>>             acquire:
>>>>>>             1. acquire write lock for all Services
>>>>>>
>>>>>>             release:
>>>>>>             1. release write lock for all Services
>>>>>>
>>>>>> In all of these examples, the lock acquiring happens from top of the
>>>>>> tree to down. This is to avoid deadlocks.
>>>>>>
>>>>>> On Mon, Oct 6, 2014 at 10:50 AM, Imesh Gunaratne <im...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Great! Thanks for the clarification Isuru!
>>>>>>>
>>>>>>> Yes I agree, I think what we can do is, identify the sub trees that
>>>>>>> will not break the consistency of the data structure and manage locks at
>>>>>>> those sub tree level.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Sun, Oct 5, 2014 at 8:07 PM, Isuru Haththotuwa <isuruh@apache.org
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hi Lahiru and Imesh,
>>>>>>>>
>>>>>>>> Thanks a lot for the input.
>>>>>>>>
>>>>>>>> What I do here is locking only the relevant sub tree of the
>>>>>>>> complete Topology tree, as locking the whole tree is rather inefficient.
>>>>>>>> For an example, when a MemberActivated event is received, we have the
>>>>>>>> cluster id of the cluster that particular member belongs to. IMHO, we only
>>>>>>>> need to acquire the write lock for that cluster , and do not need the lock
>>>>>>>> for complete Topology tree. Therefore, any other thread which needs to do
>>>>>>>> another operation on a separate sub tree (for an example, deploy a new
>>>>>>>> service, etc.) can do that concurrently.
>>>>>>>>
>>>>>>>> On Sun, Oct 5, 2014 at 12:20 AM, Lahiru Sandaruwan <
>>>>>>>> lahirus@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Isuru,
>>>>>>>>>
>>>>>>>>> Looks like a good move to improve the efficiency,
>>>>>>>>>
>>>>>>>>> On Fri, Oct 3, 2014 at 3:05 PM, Isuru Haththotuwa <
>>>>>>>>> isuruh@apache.org> wrote:
>>>>>>>>>
>>>>>>>>>> I did the initial changes, at the testing phase now. For an
>>>>>>>>>> example, if we need to add a new Service, we do not need to lock everything
>>>>>>>>>> now. We an only acquire the write lock on Services, and add the Service.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Service can be an element of a group or an app. So shouldn't we
>>>>>>>>> lock complete application if we add/modify a service? Otherwise a different
>>>>>>>>> thread might change parents/relationships otherwise.
>>>>>>>>>
>>>>>>>> AFAIK a Service gets created when we deploy a cartridge. An
>>>>>>>> application/Group can refer a service. In the case of modifying an
>>>>>>>> Application, we do need to lock the relevant clusters that belong to that
>>>>>>>> Application. I implemented that.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> So generally i think we can bring down the locking level to
>>>>>>>>> Application, but not the services. Also if we need to read any part, we
>>>>>>>>> have to get the read lock for the whole topology, such that the receiver
>>>>>>>>> get a particular snapshot of the topology as Imesh also mentioned.
>>>>>>>>>
>>>>>>>> If we need to lock the complete Topology, we can still do that,
>>>>>>>> such as in a Complete Topology event. But IMHO, if we know the exact part
>>>>>>>> (a particular Cluster, etc.) that we need to read/write, we do not need to
>>>>>>>> lock the whole Topology.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> If we need to modify a particular Service, can read lock the
>>>>>>>>>> Services and acquire the write lock on the relevant Service and do the
>>>>>>>>>> modification.
>>>>>>>>>>
>>>>>>>>> This support is there for Services, Cluster and Applications.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Wed, Oct 1, 2014 at 9:27 PM, Manula Chathurika Thantriwatte <
>>>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Isuru,
>>>>>>>>>>>
>>>>>>>>>>> +1 for the hierarchical locking approach. Using hierarchical
>>>>>>>>>>> locking we can have more benefits [1].
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> http://synapticnulship.com/blog/2013/03/08/comparison-chainlocker-vs-heirarchical-mutexes/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Oct 1, 2014 at 7:46 PM, Isuru Haththotuwa <
>>>>>>>>>>> isuruh@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Devs,
>>>>>>>>>>>>
>>>>>>>>>>>> In the current Topology implementation, we acquire read/write
>>>>>>>>>>>> locks on Topology from the root level itself. For an example, if we need to
>>>>>>>>>>>> modify a single Cluster object, we still need to acquire a write lock from
>>>>>>>>>>>> the Topology root level. But, this is a costly operation. Specially, with
>>>>>>>>>>>> Service Grouping changes, we would need to traverse through an Application.
>>>>>>>>>>>> Since an Application can be a recursive structure, it can be a time
>>>>>>>>>>>> consuming operation. in such a scenario, if we are to lock the whole
>>>>>>>>>>>> Topology, there will be many threads waiting on that lock.
>>>>>>>>>>>>
>>>>>>>>>>>> As a solution, I think we can use hierarchical locking [1]. For
>>>>>>>>>>>> an example, when you need to obtain the write lock for a particular
>>>>>>>>>>>> Application, you do not need to lock the whole tree, but can lock only that
>>>>>>>>>>>> Application itself. However, still we need to get the read locks for the
>>>>>>>>>>>> parents.
>>>>>>>>>>>>
>>>>>>>>>>>> A separate Lock tree will be maintained for the Topology.
>>>>>>>>>>>>
>>>>>>>>>>>> Please share your feedback.
>>>>>>>>>>>>
>>>>>>>>>>>> [1].
>>>>>>>>>>>> http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/2008/0801/071201hs01/071201hs01.html
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Isuru H.
>>>>>>>>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>>> Software Engineer
>>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>>
>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>>> phone : +94 772492511
>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> <http://manulachathurika.blogspot.com/>
>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>
>>>>>>>>>>> Isuru H.
>>>>>>>>>>> <http://manulachathurika.blogspot.com/>
>>>>>>>>>>> +94 716 358 048 <http://manulachathurika.blogspot.com/>*
>>>>>>>>>>> <http://wso2.com/>*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> * <http://wso2.com/>*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> --
>>>>>>>>> Lahiru Sandaruwan
>>>>>>>>> Committer and PMC member, Apache Stratos,
>>>>>>>>> Senior Software Engineer,
>>>>>>>>> WSO2 Inc., http://wso2.com
>>>>>>>>> lean.enterprise.middleware
>>>>>>>>>
>>>>>>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>>>>>>> blog: http://lahiruwrites.blogspot.com/
>>>>>>>>> twitter: http://twitter.com/lahirus
>>>>>>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>>>>>>>>> Thanks and Regards,
>>>>>>>>>
>>>>>>>>> Isuru H.
>>>>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>>>>>>>>> +94 716 358 048
>>>>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>*
>>>>>>>>> <http://wso2.com/>*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> * <http://wso2.com/>*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Imesh Gunaratne
>>>>>>>
>>>>>>> Technical Lead, WSO2
>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>
>>>>>>> --
>>>>>>> Thanks and Regards,
>>>>>>>
>>>>>>> Isuru H.
>>>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>>>
>>>>>>>
>>>>>>> * <http://wso2.com/>*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Imesh Gunaratne
>>>>>
>>>>> Technical Lead, WSO2
>>>>> Committer & PMC Member, Apache Stratos
>>>>>
>>>>> --
>>>>> Thanks and Regards,
>>>>>
>>>>> Isuru H.
>>>>> +94 716 358 048
>>>>>
>>>>> --
>>>>> <%2B94%20716%20358%20048>
>>>>> Thanks and Regards,
>>>>>
>>>>> Isuru H.
>>>>> <%2B94%20716%20358%20048>
>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>
>>>>>
>>>>> * <http://wso2.com/>*
>>>>>
>>>>>
>>>>>
>>
>>
>> --
>>
>> Gayan Gunarathne
>> Technical Lead
>> WSO2 Inc. (http://wso2.com)
>> email  : gayang@wso2.com  | mobile : +94 766819985
>>
>> --
>> <%2B94%20766819985>
>> Thanks and Regards,
>>
>> Isuru H.
>> <%2B94%20766819985>
>> +94 716 358 048 <%2B94%20766819985>* <http://wso2.com/>*
>>
>>
>> * <http://wso2.com/>*
>>
>>
>>


-- 

Gayan Gunarathne
Technical Lead
WSO2 Inc. (http://wso2.com)
email  : gayang@wso2.com  | mobile : +94 766819985

Re: [Discuss] Hierarchical Locking for Topology

Posted by Isuru Haththotuwa <is...@apache.org>.

Hi Gayan,

On Mon, Oct 13, 2014 at 7:50 PM, Gayan Gunarathne <ga...@wso2.com> wrote:

> IMO it is based on how many siblings we have in the topology which we can
> lock independently. If we have lot of occurrences where we can able to
> lock the siblings independently it will improve the performance.But again I
> think we need to careful with the dead lock scenarios.
>
I did not understand your statement exactly. Can you please explain a bit
more?

My understanding is, if CC allows only sequential access to its own
Topology model, then that will again be a bottleneck since CC is the one
who sends out the Topology events.

>
> Thanks,
> Gayan
>
> On Mon, Oct 13, 2014 at 5:50 PM, Isuru Haththotuwa <is...@apache.org>
> wrote:
>
>> Another improvement that we can do is to introduce the same hierarchical
>> locking to the Topology structure maintained in the Cloud Controller. Its
>> actually the CC who will be updating the local Topology and sending the
>> relevant events. In that case, if we have have only one write lock for the
>> whole Topology in the CC's TopologyBuilder, it will still be a bottleneck.
>> WDYT?
>>
>> On Thu, Oct 9, 2014 at 3:36 PM, Isuru Haththotuwa <is...@apache.org>
>> wrote:
>>
>>> Hi,
>>>
>>>
>>> Since the Topology is updated from the messaging component only, I
>>> removed the methods to access the write locks to an internal class. Those
>>> methods will not be exposed to outside by the messaging component. Now, the
>>> following methods to obtain read-only locks are exposed from
>>> TopologyManager:
>>>
>>>     // Topology level read locks
>>>
>>>     /**
>>>      * Acquires read lock for the Complete Topology
>>>      */
>>> *    public static void acquireReadLock() ;*
>>>
>>>     /**
>>>      * Releases read lock for the Complete Topology
>>>      */
>>> *    public static void releaseReadLock();*
>>>
>>>     // Application and Service read locks
>>>
>>>     /**
>>>      * Acquires read lock for the all Applications
>>>      */
>>> *    public static void acquireReadLockForApplications() ;*
>>>
>>>     /**
>>>      * Releases read lock for the all Applications
>>>      */
>>> *    public static void releaseReadLockForApplications();*
>>>     /**
>>>      * Acquires read lock for the all Services
>>>      */
>>> *    public static void acquireReadLockForServices();*
>>>
>>>     /**
>>>      * Releases read lock for the all Services
>>>      */
>>> *    public static void releaseReadLockForServices() ;*
>>>
>>>     /**
>>>      * Acquires read lock for a Service
>>>      *
>>>      * @param serviceName service name to acquire read lock
>>>      */
>>> *    public static void acquireReadLockForService (String serviceName) ;*
>>>
>>>     /**
>>>      * Releases read lock for a Service
>>>      *
>>>      * @param serviceName service name to release read lock
>>>      */
>>> *    public static void releaseReadLockForService (String serviceName) ;*
>>>
>>>     /**
>>>      * Acquires read lock for a Cluster. This will acquire the read lock
>>> in the following order
>>>      *      1. for the Service
>>>      *      2. for the Cluster
>>>      *
>>>      * @param serviceName service name to acquire read lock
>>>      * @param clusterId cluster id to acquire read lock
>>>      */
>>> *    public static void acquireReadLockForCluster (String serviceName,
>>> String clusterId);*
>>>
>>>     /**
>>>      * Releases read lock for a Cluster. This will release the read lock
>>> in the following order
>>>      *      1. for the Cluster
>>>      *      2. for the Service
>>>      *
>>>      * @param serviceName service name to release read lock
>>>      * @param clusterId cluster id to release read lock
>>>      */
>>> *    public static void releaseReadLockForCluster (String serviceName,
>>> String clusterId);*
>>>
>>>     /**
>>>      * Acquires read lock for the Application
>>>      *
>>>      * @param appId Application id
>>>      */
>>> *    public static void acquireReadLockForApplication (String appId) ;*
>>>
>>>     /**
>>>      * Releases read lock for the Application
>>>      *
>>>      * @param appId Application id
>>>      */
>>> *    public static void releaseReadLockForApplication (String appId);*
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Oct 6, 2014 at 2:51 PM, Imesh Gunaratne <im...@apache.org>
>>> wrote:
>>>
>>>> This looks great! As we discussed may be we could provide an interface
>>>> in the messaging component to acquire and release locks at different sub
>>>> tree levels. The whole idea is to avoid any possibilities of reading the
>>>> topology in an inconsistent state.
>>>>
>>>> Thanks
>>>>
>>>> On Mon, Oct 6, 2014 at 12:37 PM, Isuru Haththotuwa <is...@apache.org>
>>>> wrote:
>>>>
>>>>> Thanks Imesh.
>>>>>
>>>>> I have shown some examples for the new hierarchical locking approach.
>>>>> Please do let me know your feedback on this.
>>>>>
>>>>>    Acquire a write lock:
>>>>>
>>>>>    From root level, acquire read lock, and acquire a write lock only
>>>>> for the
>>>>>    relevant sub tree.
>>>>>
>>>>>    Acquire a read lock:
>>>>>
>>>>>    From root level, acquire read locks till the relevant sub tree
>>>>>
>>>>>    Examples -
>>>>>
>>>>>    Example 1: Acquiring write lock for a Cluster to modify the Cluster
>>>>> object -
>>>>>             acquiring:
>>>>>             1. acquire read lock for all Services
>>>>>             2. acquire read lock for the particular Service, to which
>>>>> the cluster belongs to
>>>>>             3. acquire write lock for the Cluster
>>>>>
>>>>>             releasing:
>>>>>             1. release write lock for the Cluster
>>>>>             2. release read lock for the particular Service
>>>>>             3. release read lock for all Services
>>>>>
>>>>>    Example 2: Acquiring write lock to add a new Cluster object -
>>>>>             acquiring:
>>>>>             1. acquire read lock for all Services
>>>>>             2. acquire write lock for the particular Service, to which
>>>>> the cluster belongs to
>>>>>
>>>>>             releasing:
>>>>>             1. release write lock for the particular Service
>>>>>             2. release read lock for all Services
>>>>>
>>>>>    Example 3: Acquiring read lock to read Cluster information
>>>>>             acquiring:
>>>>>             1. acquire read lock for all Services
>>>>>             2. acquire read lock for the particular Service, to which
>>>>> the cluster belongs to
>>>>>             3. acquire read lock for the relevant Cluster
>>>>>
>>>>>             releasing:
>>>>>             1. release read lock for the relevant Cluster
>>>>>             2. release read lock for the particular Service
>>>>>             3. release read lock for all Services
>>>>>
>>>>>    Example 4: Acquiring the write lock to add a deploy a Cartridge
>>>>> (add a new Service)
>>>>>             acquire:
>>>>>             1. acquire write lock for all Services
>>>>>
>>>>>             release:
>>>>>             1. release write lock for all Services
>>>>>
>>>>> In all of these examples, the lock acquiring happens from top of the
>>>>> tree to down. This is to avoid deadlocks.
>>>>>
>>>>> On Mon, Oct 6, 2014 at 10:50 AM, Imesh Gunaratne <im...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Great! Thanks for the clarification Isuru!
>>>>>>
>>>>>> Yes I agree, I think what we can do is, identify the sub trees that
>>>>>> will not break the consistency of the data structure and manage locks at
>>>>>> those sub tree level.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Sun, Oct 5, 2014 at 8:07 PM, Isuru Haththotuwa <is...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Lahiru and Imesh,
>>>>>>>
>>>>>>> Thanks a lot for the input.
>>>>>>>
>>>>>>> What I do here is locking only the relevant sub tree of the complete
>>>>>>> Topology tree, as locking the whole tree is rather inefficient. For an
>>>>>>> example, when a MemberActivated event is received, we have the cluster id
>>>>>>> of the cluster that particular member belongs to. IMHO, we only need to
>>>>>>> acquire the write lock for that cluster , and do not need the lock for
>>>>>>> complete Topology tree. Therefore, any other thread which needs to do
>>>>>>> another operation on a separate sub tree (for an example, deploy a new
>>>>>>> service, etc.) can do that concurrently.
>>>>>>>
>>>>>>> On Sun, Oct 5, 2014 at 12:20 AM, Lahiru Sandaruwan <lahirus@wso2.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hi Isuru,
>>>>>>>>
>>>>>>>> Looks like a good move to improve the efficiency,
>>>>>>>>
>>>>>>>> On Fri, Oct 3, 2014 at 3:05 PM, Isuru Haththotuwa <
>>>>>>>> isuruh@apache.org> wrote:
>>>>>>>>
>>>>>>>>> I did the initial changes, at the testing phase now. For an
>>>>>>>>> example, if we need to add a new Service, we do not need to lock everything
>>>>>>>>> now. We an only acquire the write lock on Services, and add the Service.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Service can be an element of a group or an app. So shouldn't we
>>>>>>>> lock complete application if we add/modify a service? Otherwise a different
>>>>>>>> thread might change parents/relationships otherwise.
>>>>>>>>
>>>>>>> AFAIK a Service gets created when we deploy a cartridge. An
>>>>>>> application/Group can refer a service. In the case of modifying an
>>>>>>> Application, we do need to lock the relevant clusters that belong to that
>>>>>>> Application. I implemented that.
>>>>>>>
>>>>>>>>
>>>>>>>> So generally i think we can bring down the locking level to
>>>>>>>> Application, but not the services. Also if we need to read any part, we
>>>>>>>> have to get the read lock for the whole topology, such that the receiver
>>>>>>>> get a particular snapshot of the topology as Imesh also mentioned.
>>>>>>>>
>>>>>>> If we need to lock the complete Topology, we can still do that, such
>>>>>>> as in a Complete Topology event. But IMHO, if we know the exact part (a
>>>>>>> particular Cluster, etc.) that we need to read/write, we do not need to
>>>>>>> lock the whole Topology.
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> If we need to modify a particular Service, can read lock the
>>>>>>>>> Services and acquire the write lock on the relevant Service and do the
>>>>>>>>> modification.
>>>>>>>>>
>>>>>>>> This support is there for Services, Cluster and Applications.
>>>>>>>>>
>>>>>>>>
>>>>>>>>> On Wed, Oct 1, 2014 at 9:27 PM, Manula Chathurika Thantriwatte <
>>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Isuru,
>>>>>>>>>>
>>>>>>>>>> +1 for the hierarchical locking approach. Using hierarchical
>>>>>>>>>> locking we can have more benefits [1].
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> http://synapticnulship.com/blog/2013/03/08/comparison-chainlocker-vs-heirarchical-mutexes/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Oct 1, 2014 at 7:46 PM, Isuru Haththotuwa <
>>>>>>>>>> isuruh@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Devs,
>>>>>>>>>>>
>>>>>>>>>>> In the current Topology implementation, we acquire read/write
>>>>>>>>>>> locks on Topology from the root level itself. For an example, if we need to
>>>>>>>>>>> modify a single Cluster object, we still need to acquire a write lock from
>>>>>>>>>>> the Topology root level. But, this is a costly operation. Specially, with
>>>>>>>>>>> Service Grouping changes, we would need to traverse through an Application.
>>>>>>>>>>> Since an Application can be a recursive structure, it can be a time
>>>>>>>>>>> consuming operation. in such a scenario, if we are to lock the whole
>>>>>>>>>>> Topology, there will be many threads waiting on that lock.
>>>>>>>>>>>
>>>>>>>>>>> As a solution, I think we can use hierarchical locking [1]. For
>>>>>>>>>>> an example, when you need to obtain the write lock for a particular
>>>>>>>>>>> Application, you do not need to lock the whole tree, but can lock only that
>>>>>>>>>>> Application itself. However, still we need to get the read locks for the
>>>>>>>>>>> parents.
>>>>>>>>>>>
>>>>>>>>>>> A separate Lock tree will be maintained for the Topology.
>>>>>>>>>>>
>>>>>>>>>>> Please share your feedback.
>>>>>>>>>>>
>>>>>>>>>>> [1].
>>>>>>>>>>> http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/2008/0801/071201hs01/071201hs01.html
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>
>>>>>>>>>>> Isuru H.
>>>>>>>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>>> Software Engineer
>>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>>
>>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>>> phone : +94 772492511
>>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> <http://manulachathurika.blogspot.com/>
>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>
>>>>>>>>>> Isuru H.
>>>>>>>>>> <http://manulachathurika.blogspot.com/>
>>>>>>>>>> +94 716 358 048 <http://manulachathurika.blogspot.com/>*
>>>>>>>>>> <http://wso2.com/>*
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> * <http://wso2.com/>*
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> --
>>>>>>>> Lahiru Sandaruwan
>>>>>>>> Committer and PMC member, Apache Stratos,
>>>>>>>> Senior Software Engineer,
>>>>>>>> WSO2 Inc., http://wso2.com
>>>>>>>> lean.enterprise.middleware
>>>>>>>>
>>>>>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>>>>>> blog: http://lahiruwrites.blogspot.com/
>>>>>>>> twitter: http://twitter.com/lahirus
>>>>>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>>>>>
>>>>>>>> --
>>>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>>>>>>>> Thanks and Regards,
>>>>>>>>
>>>>>>>> Isuru H.
>>>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>>>>>>>> +94 716 358 048
>>>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>*
>>>>>>>> <http://wso2.com/>*
>>>>>>>>
>>>>>>>>
>>>>>>>> * <http://wso2.com/>*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Imesh Gunaratne
>>>>>>
>>>>>> Technical Lead, WSO2
>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>
>>>>>> --
>>>>>> Thanks and Regards,
>>>>>>
>>>>>> Isuru H.
>>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>>
>>>>>>
>>>>>> * <http://wso2.com/>*
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> Imesh Gunaratne
>>>>
>>>> Technical Lead, WSO2
>>>> Committer & PMC Member, Apache Stratos
>>>>
>>>> --
>>>> Thanks and Regards,
>>>>
>>>> Isuru H.
>>>> +94 716 358 048
>>>>
>>>> --
>>>> <%2B94%20716%20358%20048>
>>>> Thanks and Regards,
>>>>
>>>> Isuru H.
>>>> <%2B94%20716%20358%20048>
>>>> +94 716 358 048* <http://wso2.com/>*
>>>>
>>>>
>>>> * <http://wso2.com/>*
>>>>
>>>>
>>>>
>
>
> --
>
> Gayan Gunarathne
> Technical Lead
> WSO2 Inc. (http://wso2.com)
> email  : gayang@wso2.com  | mobile : +94 766819985
>
> --
> <%2B94%20766819985>
> Thanks and Regards,
>
> Isuru H.
> <%2B94%20766819985>
> +94 716 358 048 <%2B94%20766819985>* <http://wso2.com/>*
>
>
> * <http://wso2.com/>*
>
>
>

Re: [Discuss] Hierarchical Locking for Topology

Posted by Gayan Gunarathne <ga...@wso2.com>.

IMO it is based on how many siblings we have in the topology which we can
lock independently. If we have lot of occurrences where we can able to lock
the siblings independently it will improve the performance.But again I
think we need to careful with the dead lock scenarios.

Thanks,
Gayan

On Mon, Oct 13, 2014 at 5:50 PM, Isuru Haththotuwa <is...@apache.org>
wrote:

> Another improvement that we can do is to introduce the same hierarchical
> locking to the Topology structure maintained in the Cloud Controller. Its
> actually the CC who will be updating the local Topology and sending the
> relevant events. In that case, if we have have only one write lock for the
> whole Topology in the CC's TopologyBuilder, it will still be a bottleneck.
> WDYT?
>
> On Thu, Oct 9, 2014 at 3:36 PM, Isuru Haththotuwa <is...@apache.org>
> wrote:
>
>> Hi,
>>
>>
>> Since the Topology is updated from the messaging component only, I
>> removed the methods to access the write locks to an internal class. Those
>> methods will not be exposed to outside by the messaging component. Now, the
>> following methods to obtain read-only locks are exposed from
>> TopologyManager:
>>
>>     // Topology level read locks
>>
>>     /**
>>      * Acquires read lock for the Complete Topology
>>      */
>> *    public static void acquireReadLock() ;*
>>
>>     /**
>>      * Releases read lock for the Complete Topology
>>      */
>> *    public static void releaseReadLock();*
>>
>>     // Application and Service read locks
>>
>>     /**
>>      * Acquires read lock for the all Applications
>>      */
>> *    public static void acquireReadLockForApplications() ;*
>>
>>     /**
>>      * Releases read lock for the all Applications
>>      */
>> *    public static void releaseReadLockForApplications();*
>>     /**
>>      * Acquires read lock for the all Services
>>      */
>> *    public static void acquireReadLockForServices();*
>>
>>     /**
>>      * Releases read lock for the all Services
>>      */
>> *    public static void releaseReadLockForServices() ;*
>>
>>     /**
>>      * Acquires read lock for a Service
>>      *
>>      * @param serviceName service name to acquire read lock
>>      */
>> *    public static void acquireReadLockForService (String serviceName) ;*
>>
>>     /**
>>      * Releases read lock for a Service
>>      *
>>      * @param serviceName service name to release read lock
>>      */
>> *    public static void releaseReadLockForService (String serviceName) ;*
>>
>>     /**
>>      * Acquires read lock for a Cluster. This will acquire the read lock
>> in the following order
>>      *      1. for the Service
>>      *      2. for the Cluster
>>      *
>>      * @param serviceName service name to acquire read lock
>>      * @param clusterId cluster id to acquire read lock
>>      */
>> *    public static void acquireReadLockForCluster (String serviceName,
>> String clusterId);*
>>
>>     /**
>>      * Releases read lock for a Cluster. This will release the read lock
>> in the following order
>>      *      1. for the Cluster
>>      *      2. for the Service
>>      *
>>      * @param serviceName service name to release read lock
>>      * @param clusterId cluster id to release read lock
>>      */
>> *    public static void releaseReadLockForCluster (String serviceName,
>> String clusterId);*
>>
>>     /**
>>      * Acquires read lock for the Application
>>      *
>>      * @param appId Application id
>>      */
>> *    public static void acquireReadLockForApplication (String appId) ;*
>>
>>     /**
>>      * Releases read lock for the Application
>>      *
>>      * @param appId Application id
>>      */
>> *    public static void releaseReadLockForApplication (String appId);*
>>
>>
>>
>>
>>
>> On Mon, Oct 6, 2014 at 2:51 PM, Imesh Gunaratne <im...@apache.org> wrote:
>>
>>> This looks great! As we discussed may be we could provide an interface
>>> in the messaging component to acquire and release locks at different sub
>>> tree levels. The whole idea is to avoid any possibilities of reading the
>>> topology in an inconsistent state.
>>>
>>> Thanks
>>>
>>> On Mon, Oct 6, 2014 at 12:37 PM, Isuru Haththotuwa <is...@apache.org>
>>> wrote:
>>>
>>>> Thanks Imesh.
>>>>
>>>> I have shown some examples for the new hierarchical locking approach.
>>>> Please do let me know your feedback on this.
>>>>
>>>>    Acquire a write lock:
>>>>
>>>>    From root level, acquire read lock, and acquire a write lock only
>>>> for the
>>>>    relevant sub tree.
>>>>
>>>>    Acquire a read lock:
>>>>
>>>>    From root level, acquire read locks till the relevant sub tree
>>>>
>>>>    Examples -
>>>>
>>>>    Example 1: Acquiring write lock for a Cluster to modify the Cluster
>>>> object -
>>>>             acquiring:
>>>>             1. acquire read lock for all Services
>>>>             2. acquire read lock for the particular Service, to which
>>>> the cluster belongs to
>>>>             3. acquire write lock for the Cluster
>>>>
>>>>             releasing:
>>>>             1. release write lock for the Cluster
>>>>             2. release read lock for the particular Service
>>>>             3. release read lock for all Services
>>>>
>>>>    Example 2: Acquiring write lock to add a new Cluster object -
>>>>             acquiring:
>>>>             1. acquire read lock for all Services
>>>>             2. acquire write lock for the particular Service, to which
>>>> the cluster belongs to
>>>>
>>>>             releasing:
>>>>             1. release write lock for the particular Service
>>>>             2. release read lock for all Services
>>>>
>>>>    Example 3: Acquiring read lock to read Cluster information
>>>>             acquiring:
>>>>             1. acquire read lock for all Services
>>>>             2. acquire read lock for the particular Service, to which
>>>> the cluster belongs to
>>>>             3. acquire read lock for the relevant Cluster
>>>>
>>>>             releasing:
>>>>             1. release read lock for the relevant Cluster
>>>>             2. release read lock for the particular Service
>>>>             3. release read lock for all Services
>>>>
>>>>    Example 4: Acquiring the write lock to add a deploy a Cartridge (add
>>>> a new Service)
>>>>             acquire:
>>>>             1. acquire write lock for all Services
>>>>
>>>>             release:
>>>>             1. release write lock for all Services
>>>>
>>>> In all of these examples, the lock acquiring happens from top of the
>>>> tree to down. This is to avoid deadlocks.
>>>>
>>>> On Mon, Oct 6, 2014 at 10:50 AM, Imesh Gunaratne <im...@apache.org>
>>>> wrote:
>>>>
>>>>> Great! Thanks for the clarification Isuru!
>>>>>
>>>>> Yes I agree, I think what we can do is, identify the sub trees that
>>>>> will not break the consistency of the data structure and manage locks at
>>>>> those sub tree level.
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Sun, Oct 5, 2014 at 8:07 PM, Isuru Haththotuwa <is...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi Lahiru and Imesh,
>>>>>>
>>>>>> Thanks a lot for the input.
>>>>>>
>>>>>> What I do here is locking only the relevant sub tree of the complete
>>>>>> Topology tree, as locking the whole tree is rather inefficient. For an
>>>>>> example, when a MemberActivated event is received, we have the cluster id
>>>>>> of the cluster that particular member belongs to. IMHO, we only need to
>>>>>> acquire the write lock for that cluster , and do not need the lock for
>>>>>> complete Topology tree. Therefore, any other thread which needs to do
>>>>>> another operation on a separate sub tree (for an example, deploy a new
>>>>>> service, etc.) can do that concurrently.
>>>>>>
>>>>>> On Sun, Oct 5, 2014 at 12:20 AM, Lahiru Sandaruwan <la...@wso2.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Isuru,
>>>>>>>
>>>>>>> Looks like a good move to improve the efficiency,
>>>>>>>
>>>>>>> On Fri, Oct 3, 2014 at 3:05 PM, Isuru Haththotuwa <isuruh@apache.org
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> I did the initial changes, at the testing phase now. For an
>>>>>>>> example, if we need to add a new Service, we do not need to lock everything
>>>>>>>> now. We an only acquire the write lock on Services, and add the Service.
>>>>>>>>
>>>>>>>
>>>>>>> Service can be an element of a group or an app. So shouldn't we lock
>>>>>>> complete application if we add/modify a service? Otherwise a different
>>>>>>> thread might change parents/relationships otherwise.
>>>>>>>
>>>>>> AFAIK a Service gets created when we deploy a cartridge. An
>>>>>> application/Group can refer a service. In the case of modifying an
>>>>>> Application, we do need to lock the relevant clusters that belong to that
>>>>>> Application. I implemented that.
>>>>>>
>>>>>>>
>>>>>>> So generally i think we can bring down the locking level to
>>>>>>> Application, but not the services. Also if we need to read any part, we
>>>>>>> have to get the read lock for the whole topology, such that the receiver
>>>>>>> get a particular snapshot of the topology as Imesh also mentioned.
>>>>>>>
>>>>>> If we need to lock the complete Topology, we can still do that, such
>>>>>> as in a Complete Topology event. But IMHO, if we know the exact part (a
>>>>>> particular Cluster, etc.) that we need to read/write, we do not need to
>>>>>> lock the whole Topology.
>>>>>>
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> If we need to modify a particular Service, can read lock the
>>>>>>>> Services and acquire the write lock on the relevant Service and do the
>>>>>>>> modification.
>>>>>>>>
>>>>>>> This support is there for Services, Cluster and Applications.
>>>>>>>>
>>>>>>>
>>>>>>>> On Wed, Oct 1, 2014 at 9:27 PM, Manula Chathurika Thantriwatte <
>>>>>>>> manulac@wso2.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Isuru,
>>>>>>>>>
>>>>>>>>> +1 for the hierarchical locking approach. Using hierarchical
>>>>>>>>> locking we can have more benefits [1].
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> http://synapticnulship.com/blog/2013/03/08/comparison-chainlocker-vs-heirarchical-mutexes/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Oct 1, 2014 at 7:46 PM, Isuru Haththotuwa <
>>>>>>>>> isuruh@apache.org> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Devs,
>>>>>>>>>>
>>>>>>>>>> In the current Topology implementation, we acquire read/write
>>>>>>>>>> locks on Topology from the root level itself. For an example, if we need to
>>>>>>>>>> modify a single Cluster object, we still need to acquire a write lock from
>>>>>>>>>> the Topology root level. But, this is a costly operation. Specially, with
>>>>>>>>>> Service Grouping changes, we would need to traverse through an Application.
>>>>>>>>>> Since an Application can be a recursive structure, it can be a time
>>>>>>>>>> consuming operation. in such a scenario, if we are to lock the whole
>>>>>>>>>> Topology, there will be many threads waiting on that lock.
>>>>>>>>>>
>>>>>>>>>> As a solution, I think we can use hierarchical locking [1]. For
>>>>>>>>>> an example, when you need to obtain the write lock for a particular
>>>>>>>>>> Application, you do not need to lock the whole tree, but can lock only that
>>>>>>>>>> Application itself. However, still we need to get the read locks for the
>>>>>>>>>> parents.
>>>>>>>>>>
>>>>>>>>>> A separate Lock tree will be maintained for the Topology.
>>>>>>>>>>
>>>>>>>>>> Please share your feedback.
>>>>>>>>>>
>>>>>>>>>> [1].
>>>>>>>>>> http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/2008/0801/071201hs01/071201hs01.html
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>
>>>>>>>>>> Isuru H.
>>>>>>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>>> Software Engineer
>>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>>> lean . enterprise . middleware
>>>>>>>>>
>>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>>> phone : +94 772492511
>>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> <http://manulachathurika.blogspot.com/>
>>>>>>>>> Thanks and Regards,
>>>>>>>>>
>>>>>>>>> Isuru H.
>>>>>>>>> <http://manulachathurika.blogspot.com/>
>>>>>>>>> +94 716 358 048 <http://manulachathurika.blogspot.com/>*
>>>>>>>>> <http://wso2.com/>*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> * <http://wso2.com/>*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> --
>>>>>>> Lahiru Sandaruwan
>>>>>>> Committer and PMC member, Apache Stratos,
>>>>>>> Senior Software Engineer,
>>>>>>> WSO2 Inc., http://wso2.com
>>>>>>> lean.enterprise.middleware
>>>>>>>
>>>>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>>>>> blog: http://lahiruwrites.blogspot.com/
>>>>>>> twitter: http://twitter.com/lahirus
>>>>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>>>>
>>>>>>> --
>>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>>>>>>> Thanks and Regards,
>>>>>>>
>>>>>>> Isuru H.
>>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>>>>>>> +94 716 358 048
>>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>*
>>>>>>> <http://wso2.com/>*
>>>>>>>
>>>>>>>
>>>>>>> * <http://wso2.com/>*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Imesh Gunaratne
>>>>>
>>>>> Technical Lead, WSO2
>>>>> Committer & PMC Member, Apache Stratos
>>>>>
>>>>> --
>>>>> Thanks and Regards,
>>>>>
>>>>> Isuru H.
>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>
>>>>>
>>>>> * <http://wso2.com/>*
>>>>>
>>>>>
>>>>>
>>>
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>> --
>>> Thanks and Regards,
>>>
>>> Isuru H.
>>> +94 716 358 048
>>>
>>> --
>>> <%2B94%20716%20358%20048>
>>> Thanks and Regards,
>>>
>>> Isuru H.
>>> <%2B94%20716%20358%20048>
>>> +94 716 358 048* <http://wso2.com/>*
>>>
>>>
>>> * <http://wso2.com/>*
>>>
>>>
>>>


-- 

Gayan Gunarathne
Technical Lead
WSO2 Inc. (http://wso2.com)
email  : gayang@wso2.com  | mobile : +94 766819985

Re: [Discuss] Hierarchical Locking for Topology

Posted by Isuru Haththotuwa <is...@apache.org>.

Another improvement that we can do is to introduce the same hierarchical
locking to the Topology structure maintained in the Cloud Controller. Its
actually the CC who will be updating the local Topology and sending the
relevant events. In that case, if we have have only one write lock for the
whole Topology in the CC's TopologyBuilder, it will still be a bottleneck.
WDYT?

On Thu, Oct 9, 2014 at 3:36 PM, Isuru Haththotuwa <is...@apache.org> wrote:

> Hi,
>
>
> Since the Topology is updated from the messaging component only, I removed
> the methods to access the write locks to an internal class. Those methods
> will not be exposed to outside by the messaging component. Now, the
> following methods to obtain read-only locks are exposed from
> TopologyManager:
>
>     // Topology level read locks
>
>     /**
>      * Acquires read lock for the Complete Topology
>      */
> *    public static void acquireReadLock() ;*
>
>     /**
>      * Releases read lock for the Complete Topology
>      */
> *    public static void releaseReadLock();*
>
>     // Application and Service read locks
>
>     /**
>      * Acquires read lock for the all Applications
>      */
> *    public static void acquireReadLockForApplications() ;*
>
>     /**
>      * Releases read lock for the all Applications
>      */
> *    public static void releaseReadLockForApplications();*
>     /**
>      * Acquires read lock for the all Services
>      */
> *    public static void acquireReadLockForServices();*
>
>     /**
>      * Releases read lock for the all Services
>      */
> *    public static void releaseReadLockForServices() ;*
>
>     /**
>      * Acquires read lock for a Service
>      *
>      * @param serviceName service name to acquire read lock
>      */
> *    public static void acquireReadLockForService (String serviceName) ;*
>
>     /**
>      * Releases read lock for a Service
>      *
>      * @param serviceName service name to release read lock
>      */
> *    public static void releaseReadLockForService (String serviceName) ;*
>
>     /**
>      * Acquires read lock for a Cluster. This will acquire the read lock
> in the following order
>      *      1. for the Service
>      *      2. for the Cluster
>      *
>      * @param serviceName service name to acquire read lock
>      * @param clusterId cluster id to acquire read lock
>      */
> *    public static void acquireReadLockForCluster (String serviceName,
> String clusterId);*
>
>     /**
>      * Releases read lock for a Cluster. This will release the read lock
> in the following order
>      *      1. for the Cluster
>      *      2. for the Service
>      *
>      * @param serviceName service name to release read lock
>      * @param clusterId cluster id to release read lock
>      */
> *    public static void releaseReadLockForCluster (String serviceName,
> String clusterId);*
>
>     /**
>      * Acquires read lock for the Application
>      *
>      * @param appId Application id
>      */
> *    public static void acquireReadLockForApplication (String appId) ;*
>
>     /**
>      * Releases read lock for the Application
>      *
>      * @param appId Application id
>      */
> *    public static void releaseReadLockForApplication (String appId);*
>
>
>
>
>
> On Mon, Oct 6, 2014 at 2:51 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> This looks great! As we discussed may be we could provide an interface in
>> the messaging component to acquire and release locks at different sub tree
>> levels. The whole idea is to avoid any possibilities of reading the
>> topology in an inconsistent state.
>>
>> Thanks
>>
>> On Mon, Oct 6, 2014 at 12:37 PM, Isuru Haththotuwa <is...@apache.org>
>> wrote:
>>
>>> Thanks Imesh.
>>>
>>> I have shown some examples for the new hierarchical locking approach.
>>> Please do let me know your feedback on this.
>>>
>>>    Acquire a write lock:
>>>
>>>    From root level, acquire read lock, and acquire a write lock only for
>>> the
>>>    relevant sub tree.
>>>
>>>    Acquire a read lock:
>>>
>>>    From root level, acquire read locks till the relevant sub tree
>>>
>>>    Examples -
>>>
>>>    Example 1: Acquiring write lock for a Cluster to modify the Cluster
>>> object -
>>>             acquiring:
>>>             1. acquire read lock for all Services
>>>             2. acquire read lock for the particular Service, to which
>>> the cluster belongs to
>>>             3. acquire write lock for the Cluster
>>>
>>>             releasing:
>>>             1. release write lock for the Cluster
>>>             2. release read lock for the particular Service
>>>             3. release read lock for all Services
>>>
>>>    Example 2: Acquiring write lock to add a new Cluster object -
>>>             acquiring:
>>>             1. acquire read lock for all Services
>>>             2. acquire write lock for the particular Service, to which
>>> the cluster belongs to
>>>
>>>             releasing:
>>>             1. release write lock for the particular Service
>>>             2. release read lock for all Services
>>>
>>>    Example 3: Acquiring read lock to read Cluster information
>>>             acquiring:
>>>             1. acquire read lock for all Services
>>>             2. acquire read lock for the particular Service, to which
>>> the cluster belongs to
>>>             3. acquire read lock for the relevant Cluster
>>>
>>>             releasing:
>>>             1. release read lock for the relevant Cluster
>>>             2. release read lock for the particular Service
>>>             3. release read lock for all Services
>>>
>>>    Example 4: Acquiring the write lock to add a deploy a Cartridge (add
>>> a new Service)
>>>             acquire:
>>>             1. acquire write lock for all Services
>>>
>>>             release:
>>>             1. release write lock for all Services
>>>
>>> In all of these examples, the lock acquiring happens from top of the
>>> tree to down. This is to avoid deadlocks.
>>>
>>> On Mon, Oct 6, 2014 at 10:50 AM, Imesh Gunaratne <im...@apache.org>
>>> wrote:
>>>
>>>> Great! Thanks for the clarification Isuru!
>>>>
>>>> Yes I agree, I think what we can do is, identify the sub trees that
>>>> will not break the consistency of the data structure and manage locks at
>>>> those sub tree level.
>>>>
>>>> Thanks
>>>>
>>>> On Sun, Oct 5, 2014 at 8:07 PM, Isuru Haththotuwa <is...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Lahiru and Imesh,
>>>>>
>>>>> Thanks a lot for the input.
>>>>>
>>>>> What I do here is locking only the relevant sub tree of the complete
>>>>> Topology tree, as locking the whole tree is rather inefficient. For an
>>>>> example, when a MemberActivated event is received, we have the cluster id
>>>>> of the cluster that particular member belongs to. IMHO, we only need to
>>>>> acquire the write lock for that cluster , and do not need the lock for
>>>>> complete Topology tree. Therefore, any other thread which needs to do
>>>>> another operation on a separate sub tree (for an example, deploy a new
>>>>> service, etc.) can do that concurrently.
>>>>>
>>>>> On Sun, Oct 5, 2014 at 12:20 AM, Lahiru Sandaruwan <la...@wso2.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Isuru,
>>>>>>
>>>>>> Looks like a good move to improve the efficiency,
>>>>>>
>>>>>> On Fri, Oct 3, 2014 at 3:05 PM, Isuru Haththotuwa <is...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> I did the initial changes, at the testing phase now. For an example,
>>>>>>> if we need to add a new Service, we do not need to lock everything now. We
>>>>>>> an only acquire the write lock on Services, and add the Service.
>>>>>>>
>>>>>>
>>>>>> Service can be an element of a group or an app. So shouldn't we lock
>>>>>> complete application if we add/modify a service? Otherwise a different
>>>>>> thread might change parents/relationships otherwise.
>>>>>>
>>>>> AFAIK a Service gets created when we deploy a cartridge. An
>>>>> application/Group can refer a service. In the case of modifying an
>>>>> Application, we do need to lock the relevant clusters that belong to that
>>>>> Application. I implemented that.
>>>>>
>>>>>>
>>>>>> So generally i think we can bring down the locking level to
>>>>>> Application, but not the services. Also if we need to read any part, we
>>>>>> have to get the read lock for the whole topology, such that the receiver
>>>>>> get a particular snapshot of the topology as Imesh also mentioned.
>>>>>>
>>>>> If we need to lock the complete Topology, we can still do that, such
>>>>> as in a Complete Topology event. But IMHO, if we know the exact part (a
>>>>> particular Cluster, etc.) that we need to read/write, we do not need to
>>>>> lock the whole Topology.
>>>>>
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> If we need to modify a particular Service, can read lock the Services
>>>>>>> and acquire the write lock on the relevant Service and do the modification.
>>>>>>>
>>>>>> This support is there for Services, Cluster and Applications.
>>>>>>>
>>>>>>
>>>>>>> On Wed, Oct 1, 2014 at 9:27 PM, Manula Chathurika Thantriwatte <
>>>>>>> manulac@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi Isuru,
>>>>>>>>
>>>>>>>> +1 for the hierarchical locking approach. Using hierarchical
>>>>>>>> locking we can have more benefits [1].
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> http://synapticnulship.com/blog/2013/03/08/comparison-chainlocker-vs-heirarchical-mutexes/
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Oct 1, 2014 at 7:46 PM, Isuru Haththotuwa <
>>>>>>>> isuruh@apache.org> wrote:
>>>>>>>>
>>>>>>>>> Hi Devs,
>>>>>>>>>
>>>>>>>>> In the current Topology implementation, we acquire read/write
>>>>>>>>> locks on Topology from the root level itself. For an example, if we need to
>>>>>>>>> modify a single Cluster object, we still need to acquire a write lock from
>>>>>>>>> the Topology root level. But, this is a costly operation. Specially, with
>>>>>>>>> Service Grouping changes, we would need to traverse through an Application.
>>>>>>>>> Since an Application can be a recursive structure, it can be a time
>>>>>>>>> consuming operation. in such a scenario, if we are to lock the whole
>>>>>>>>> Topology, there will be many threads waiting on that lock.
>>>>>>>>>
>>>>>>>>> As a solution, I think we can use hierarchical locking [1]. For an
>>>>>>>>> example, when you need to obtain the write lock for a particular
>>>>>>>>> Application, you do not need to lock the whole tree, but can lock only that
>>>>>>>>> Application itself. However, still we need to get the read locks for the
>>>>>>>>> parents.
>>>>>>>>>
>>>>>>>>> A separate Lock tree will be maintained for the Topology.
>>>>>>>>>
>>>>>>>>> Please share your feedback.
>>>>>>>>>
>>>>>>>>> [1].
>>>>>>>>> http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/2008/0801/071201hs01/071201hs01.html
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Thanks and Regards,
>>>>>>>>>
>>>>>>>>> Isuru H.
>>>>>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Manula Chathurika Thantriwatte
>>>>>>>> Software Engineer
>>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>>> lean . enterprise . middleware
>>>>>>>>
>>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>>> phone : +94 772492511
>>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>>
>>>>>>>> --
>>>>>>>> <http://manulachathurika.blogspot.com/>
>>>>>>>> Thanks and Regards,
>>>>>>>>
>>>>>>>> Isuru H.
>>>>>>>> <http://manulachathurika.blogspot.com/>
>>>>>>>> +94 716 358 048 <http://manulachathurika.blogspot.com/>*
>>>>>>>> <http://wso2.com/>*
>>>>>>>>
>>>>>>>>
>>>>>>>> * <http://wso2.com/>*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> --
>>>>>> Lahiru Sandaruwan
>>>>>> Committer and PMC member, Apache Stratos,
>>>>>> Senior Software Engineer,
>>>>>> WSO2 Inc., http://wso2.com
>>>>>> lean.enterprise.middleware
>>>>>>
>>>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>>>> blog: http://lahiruwrites.blogspot.com/
>>>>>> twitter: http://twitter.com/lahirus
>>>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>>>
>>>>>> --
>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>>>>>> Thanks and Regards,
>>>>>>
>>>>>> Isuru H.
>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>>>>>> +94 716 358 048
>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>*
>>>>>> <http://wso2.com/>*
>>>>>>
>>>>>>
>>>>>> * <http://wso2.com/>*
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> Imesh Gunaratne
>>>>
>>>> Technical Lead, WSO2
>>>> Committer & PMC Member, Apache Stratos
>>>>
>>>> --
>>>> Thanks and Regards,
>>>>
>>>> Isuru H.
>>>> +94 716 358 048* <http://wso2.com/>*
>>>>
>>>>
>>>> * <http://wso2.com/>*
>>>>
>>>>
>>>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>> --
>> Thanks and Regards,
>>
>> Isuru H.
>> +94 716 358 048
>>
>> --
>> Thanks and Regards,
>>
>> Isuru H.
>> +94 716 358 048* <http://wso2.com/>*
>>
>>
>> * <http://wso2.com/>*
>>
>>
>>

Re: [Discuss] Hierarchical Locking for Topology

Posted by Isuru Haththotuwa <is...@apache.org>.

Hi,


Since the Topology is updated from the messaging component only, I removed
the methods to access the write locks to an internal class. Those methods
will not be exposed to outside by the messaging component. Now, the
following methods to obtain read-only locks are exposed from
TopologyManager:

    // Topology level read locks

    /**
     * Acquires read lock for the Complete Topology
     */
*    public static void acquireReadLock() ;*

    /**
     * Releases read lock for the Complete Topology
     */
*    public static void releaseReadLock();*

    // Application and Service read locks

    /**
     * Acquires read lock for the all Applications
     */
*    public static void acquireReadLockForApplications() ;*

    /**
     * Releases read lock for the all Applications
     */
*    public static void releaseReadLockForApplications();*
    /**
     * Acquires read lock for the all Services
     */
*    public static void acquireReadLockForServices();*

    /**
     * Releases read lock for the all Services
     */
*    public static void releaseReadLockForServices() ;*

    /**
     * Acquires read lock for a Service
     *
     * @param serviceName service name to acquire read lock
     */
*    public static void acquireReadLockForService (String serviceName) ;*

    /**
     * Releases read lock for a Service
     *
     * @param serviceName service name to release read lock
     */
*    public static void releaseReadLockForService (String serviceName) ;*

    /**
     * Acquires read lock for a Cluster. This will acquire the read lock in
the following order
     *      1. for the Service
     *      2. for the Cluster
     *
     * @param serviceName service name to acquire read lock
     * @param clusterId cluster id to acquire read lock
     */
*    public static void acquireReadLockForCluster (String serviceName,
String clusterId);*

    /**
     * Releases read lock for a Cluster. This will release the read lock in
the following order
     *      1. for the Cluster
     *      2. for the Service
     *
     * @param serviceName service name to release read lock
     * @param clusterId cluster id to release read lock
     */
*    public static void releaseReadLockForCluster (String serviceName,
String clusterId);*

    /**
     * Acquires read lock for the Application
     *
     * @param appId Application id
     */
*    public static void acquireReadLockForApplication (String appId) ;*

    /**
     * Releases read lock for the Application
     *
     * @param appId Application id
     */
*    public static void releaseReadLockForApplication (String appId);*




On Mon, Oct 6, 2014 at 2:51 PM, Imesh Gunaratne <im...@apache.org> wrote:

> This looks great! As we discussed may be we could provide an interface in
> the messaging component to acquire and release locks at different sub tree
> levels. The whole idea is to avoid any possibilities of reading the
> topology in an inconsistent state.
>
> Thanks
>
> On Mon, Oct 6, 2014 at 12:37 PM, Isuru Haththotuwa <is...@apache.org>
> wrote:
>
>> Thanks Imesh.
>>
>> I have shown some examples for the new hierarchical locking approach.
>> Please do let me know your feedback on this.
>>
>>    Acquire a write lock:
>>
>>    From root level, acquire read lock, and acquire a write lock only for
>> the
>>    relevant sub tree.
>>
>>    Acquire a read lock:
>>
>>    From root level, acquire read locks till the relevant sub tree
>>
>>    Examples -
>>
>>    Example 1: Acquiring write lock for a Cluster to modify the Cluster
>> object -
>>             acquiring:
>>             1. acquire read lock for all Services
>>             2. acquire read lock for the particular Service, to which the
>> cluster belongs to
>>             3. acquire write lock for the Cluster
>>
>>             releasing:
>>             1. release write lock for the Cluster
>>             2. release read lock for the particular Service
>>             3. release read lock for all Services
>>
>>    Example 2: Acquiring write lock to add a new Cluster object -
>>             acquiring:
>>             1. acquire read lock for all Services
>>             2. acquire write lock for the particular Service, to which
>> the cluster belongs to
>>
>>             releasing:
>>             1. release write lock for the particular Service
>>             2. release read lock for all Services
>>
>>    Example 3: Acquiring read lock to read Cluster information
>>             acquiring:
>>             1. acquire read lock for all Services
>>             2. acquire read lock for the particular Service, to which the
>> cluster belongs to
>>             3. acquire read lock for the relevant Cluster
>>
>>             releasing:
>>             1. release read lock for the relevant Cluster
>>             2. release read lock for the particular Service
>>             3. release read lock for all Services
>>
>>    Example 4: Acquiring the write lock to add a deploy a Cartridge (add a
>> new Service)
>>             acquire:
>>             1. acquire write lock for all Services
>>
>>             release:
>>             1. release write lock for all Services
>>
>> In all of these examples, the lock acquiring happens from top of the tree
>> to down. This is to avoid deadlocks.
>>
>> On Mon, Oct 6, 2014 at 10:50 AM, Imesh Gunaratne <im...@apache.org>
>> wrote:
>>
>>> Great! Thanks for the clarification Isuru!
>>>
>>> Yes I agree, I think what we can do is, identify the sub trees that will
>>> not break the consistency of the data structure and manage locks at those
>>> sub tree level.
>>>
>>> Thanks
>>>
>>> On Sun, Oct 5, 2014 at 8:07 PM, Isuru Haththotuwa <is...@apache.org>
>>> wrote:
>>>
>>>> Hi Lahiru and Imesh,
>>>>
>>>> Thanks a lot for the input.
>>>>
>>>> What I do here is locking only the relevant sub tree of the complete
>>>> Topology tree, as locking the whole tree is rather inefficient. For an
>>>> example, when a MemberActivated event is received, we have the cluster id
>>>> of the cluster that particular member belongs to. IMHO, we only need to
>>>> acquire the write lock for that cluster , and do not need the lock for
>>>> complete Topology tree. Therefore, any other thread which needs to do
>>>> another operation on a separate sub tree (for an example, deploy a new
>>>> service, etc.) can do that concurrently.
>>>>
>>>> On Sun, Oct 5, 2014 at 12:20 AM, Lahiru Sandaruwan <la...@wso2.com>
>>>> wrote:
>>>>
>>>>> Hi Isuru,
>>>>>
>>>>> Looks like a good move to improve the efficiency,
>>>>>
>>>>> On Fri, Oct 3, 2014 at 3:05 PM, Isuru Haththotuwa <is...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> I did the initial changes, at the testing phase now. For an example,
>>>>>> if we need to add a new Service, we do not need to lock everything now. We
>>>>>> an only acquire the write lock on Services, and add the Service.
>>>>>>
>>>>>
>>>>> Service can be an element of a group or an app. So shouldn't we lock
>>>>> complete application if we add/modify a service? Otherwise a different
>>>>> thread might change parents/relationships otherwise.
>>>>>
>>>> AFAIK a Service gets created when we deploy a cartridge. An
>>>> application/Group can refer a service. In the case of modifying an
>>>> Application, we do need to lock the relevant clusters that belong to that
>>>> Application. I implemented that.
>>>>
>>>>>
>>>>> So generally i think we can bring down the locking level to
>>>>> Application, but not the services. Also if we need to read any part, we
>>>>> have to get the read lock for the whole topology, such that the receiver
>>>>> get a particular snapshot of the topology as Imesh also mentioned.
>>>>>
>>>> If we need to lock the complete Topology, we can still do that, such as
>>>> in a Complete Topology event. But IMHO, if we know the exact part (a
>>>> particular Cluster, etc.) that we need to read/write, we do not need to
>>>> lock the whole Topology.
>>>>
>>>>>
>>>>> Thanks.
>>>>>
>>>>> If we need to modify a particular Service, can read lock the Services
>>>>>> and acquire the write lock on the relevant Service and do the modification.
>>>>>>
>>>>> This support is there for Services, Cluster and Applications.
>>>>>>
>>>>>
>>>>>> On Wed, Oct 1, 2014 at 9:27 PM, Manula Chathurika Thantriwatte <
>>>>>> manulac@wso2.com> wrote:
>>>>>>
>>>>>>> Hi Isuru,
>>>>>>>
>>>>>>> +1 for the hierarchical locking approach. Using hierarchical locking
>>>>>>> we can have more benefits [1].
>>>>>>>
>>>>>>> [1]
>>>>>>> http://synapticnulship.com/blog/2013/03/08/comparison-chainlocker-vs-heirarchical-mutexes/
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Oct 1, 2014 at 7:46 PM, Isuru Haththotuwa <isuruh@apache.org
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hi Devs,
>>>>>>>>
>>>>>>>> In the current Topology implementation, we acquire read/write locks
>>>>>>>> on Topology from the root level itself. For an example, if we need to
>>>>>>>> modify a single Cluster object, we still need to acquire a write lock from
>>>>>>>> the Topology root level. But, this is a costly operation. Specially, with
>>>>>>>> Service Grouping changes, we would need to traverse through an Application.
>>>>>>>> Since an Application can be a recursive structure, it can be a time
>>>>>>>> consuming operation. in such a scenario, if we are to lock the whole
>>>>>>>> Topology, there will be many threads waiting on that lock.
>>>>>>>>
>>>>>>>> As a solution, I think we can use hierarchical locking [1]. For an
>>>>>>>> example, when you need to obtain the write lock for a particular
>>>>>>>> Application, you do not need to lock the whole tree, but can lock only that
>>>>>>>> Application itself. However, still we need to get the read locks for the
>>>>>>>> parents.
>>>>>>>>
>>>>>>>> A separate Lock tree will be maintained for the Topology.
>>>>>>>>
>>>>>>>> Please share your feedback.
>>>>>>>>
>>>>>>>> [1].
>>>>>>>> http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/2008/0801/071201hs01/071201hs01.html
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks and Regards,
>>>>>>>>
>>>>>>>> Isuru H.
>>>>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Manula Chathurika Thantriwatte
>>>>>>> Software Engineer
>>>>>>> WSO2 Inc. : http://wso2.com
>>>>>>> lean . enterprise . middleware
>>>>>>>
>>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>>> phone : +94 772492511
>>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>>
>>>>>>> --
>>>>>>> <http://manulachathurika.blogspot.com/>
>>>>>>> Thanks and Regards,
>>>>>>>
>>>>>>> Isuru H.
>>>>>>> <http://manulachathurika.blogspot.com/>
>>>>>>> +94 716 358 048 <http://manulachathurika.blogspot.com/>*
>>>>>>> <http://wso2.com/>*
>>>>>>>
>>>>>>>
>>>>>>> * <http://wso2.com/>*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --
>>>>> Lahiru Sandaruwan
>>>>> Committer and PMC member, Apache Stratos,
>>>>> Senior Software Engineer,
>>>>> WSO2 Inc., http://wso2.com
>>>>> lean.enterprise.middleware
>>>>>
>>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>>> blog: http://lahiruwrites.blogspot.com/
>>>>> twitter: http://twitter.com/lahirus
>>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>>
>>>>> --
>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>>>>> Thanks and Regards,
>>>>>
>>>>> Isuru H.
>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>>>>> +94 716 358 048
>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>*
>>>>> <http://wso2.com/>*
>>>>>
>>>>>
>>>>> * <http://wso2.com/>*
>>>>>
>>>>>
>>>>>
>>>
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>> --
>>> Thanks and Regards,
>>>
>>> Isuru H.
>>> +94 716 358 048* <http://wso2.com/>*
>>>
>>>
>>> * <http://wso2.com/>*
>>>
>>>
>>>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
> --
> Thanks and Regards,
>
> Isuru H.
> +94 716 358 048* <http://wso2.com/>*
>
>
> * <http://wso2.com/>*
>
>
>

Re: [Discuss] Hierarchical Locking for Topology

Posted by Imesh Gunaratne <im...@apache.org>.

This looks great! As we discussed may be we could provide an interface in
the messaging component to acquire and release locks at different sub tree
levels. The whole idea is to avoid any possibilities of reading the
topology in an inconsistent state.

Thanks

On Mon, Oct 6, 2014 at 12:37 PM, Isuru Haththotuwa <is...@apache.org>
wrote:

> Thanks Imesh.
>
> I have shown some examples for the new hierarchical locking approach.
> Please do let me know your feedback on this.
>
>    Acquire a write lock:
>
>    From root level, acquire read lock, and acquire a write lock only for
> the
>    relevant sub tree.
>
>    Acquire a read lock:
>
>    From root level, acquire read locks till the relevant sub tree
>
>    Examples -
>
>    Example 1: Acquiring write lock for a Cluster to modify the Cluster
> object -
>             acquiring:
>             1. acquire read lock for all Services
>             2. acquire read lock for the particular Service, to which the
> cluster belongs to
>             3. acquire write lock for the Cluster
>
>             releasing:
>             1. release write lock for the Cluster
>             2. release read lock for the particular Service
>             3. release read lock for all Services
>
>    Example 2: Acquiring write lock to add a new Cluster object -
>             acquiring:
>             1. acquire read lock for all Services
>             2. acquire write lock for the particular Service, to which the
> cluster belongs to
>
>             releasing:
>             1. release write lock for the particular Service
>             2. release read lock for all Services
>
>    Example 3: Acquiring read lock to read Cluster information
>             acquiring:
>             1. acquire read lock for all Services
>             2. acquire read lock for the particular Service, to which the
> cluster belongs to
>             3. acquire read lock for the relevant Cluster
>
>             releasing:
>             1. release read lock for the relevant Cluster
>             2. release read lock for the particular Service
>             3. release read lock for all Services
>
>    Example 4: Acquiring the write lock to add a deploy a Cartridge (add a
> new Service)
>             acquire:
>             1. acquire write lock for all Services
>
>             release:
>             1. release write lock for all Services
>
> In all of these examples, the lock acquiring happens from top of the tree
> to down. This is to avoid deadlocks.
>
> On Mon, Oct 6, 2014 at 10:50 AM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> Great! Thanks for the clarification Isuru!
>>
>> Yes I agree, I think what we can do is, identify the sub trees that will
>> not break the consistency of the data structure and manage locks at those
>> sub tree level.
>>
>> Thanks
>>
>> On Sun, Oct 5, 2014 at 8:07 PM, Isuru Haththotuwa <is...@apache.org>
>> wrote:
>>
>>> Hi Lahiru and Imesh,
>>>
>>> Thanks a lot for the input.
>>>
>>> What I do here is locking only the relevant sub tree of the complete
>>> Topology tree, as locking the whole tree is rather inefficient. For an
>>> example, when a MemberActivated event is received, we have the cluster id
>>> of the cluster that particular member belongs to. IMHO, we only need to
>>> acquire the write lock for that cluster , and do not need the lock for
>>> complete Topology tree. Therefore, any other thread which needs to do
>>> another operation on a separate sub tree (for an example, deploy a new
>>> service, etc.) can do that concurrently.
>>>
>>> On Sun, Oct 5, 2014 at 12:20 AM, Lahiru Sandaruwan <la...@wso2.com>
>>> wrote:
>>>
>>>> Hi Isuru,
>>>>
>>>> Looks like a good move to improve the efficiency,
>>>>
>>>> On Fri, Oct 3, 2014 at 3:05 PM, Isuru Haththotuwa <is...@apache.org>
>>>> wrote:
>>>>
>>>>> I did the initial changes, at the testing phase now. For an example,
>>>>> if we need to add a new Service, we do not need to lock everything now. We
>>>>> an only acquire the write lock on Services, and add the Service.
>>>>>
>>>>
>>>> Service can be an element of a group or an app. So shouldn't we lock
>>>> complete application if we add/modify a service? Otherwise a different
>>>> thread might change parents/relationships otherwise.
>>>>
>>> AFAIK a Service gets created when we deploy a cartridge. An
>>> application/Group can refer a service. In the case of modifying an
>>> Application, we do need to lock the relevant clusters that belong to that
>>> Application. I implemented that.
>>>
>>>>
>>>> So generally i think we can bring down the locking level to
>>>> Application, but not the services. Also if we need to read any part, we
>>>> have to get the read lock for the whole topology, such that the receiver
>>>> get a particular snapshot of the topology as Imesh also mentioned.
>>>>
>>> If we need to lock the complete Topology, we can still do that, such as
>>> in a Complete Topology event. But IMHO, if we know the exact part (a
>>> particular Cluster, etc.) that we need to read/write, we do not need to
>>> lock the whole Topology.
>>>
>>>>
>>>> Thanks.
>>>>
>>>> If we need to modify a particular Service, can read lock the Services
>>>>> and acquire the write lock on the relevant Service and do the modification.
>>>>>
>>>> This support is there for Services, Cluster and Applications.
>>>>>
>>>>
>>>>> On Wed, Oct 1, 2014 at 9:27 PM, Manula Chathurika Thantriwatte <
>>>>> manulac@wso2.com> wrote:
>>>>>
>>>>>> Hi Isuru,
>>>>>>
>>>>>> +1 for the hierarchical locking approach. Using hierarchical locking
>>>>>> we can have more benefits [1].
>>>>>>
>>>>>> [1]
>>>>>> http://synapticnulship.com/blog/2013/03/08/comparison-chainlocker-vs-heirarchical-mutexes/
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 1, 2014 at 7:46 PM, Isuru Haththotuwa <is...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Devs,
>>>>>>>
>>>>>>> In the current Topology implementation, we acquire read/write locks
>>>>>>> on Topology from the root level itself. For an example, if we need to
>>>>>>> modify a single Cluster object, we still need to acquire a write lock from
>>>>>>> the Topology root level. But, this is a costly operation. Specially, with
>>>>>>> Service Grouping changes, we would need to traverse through an Application.
>>>>>>> Since an Application can be a recursive structure, it can be a time
>>>>>>> consuming operation. in such a scenario, if we are to lock the whole
>>>>>>> Topology, there will be many threads waiting on that lock.
>>>>>>>
>>>>>>> As a solution, I think we can use hierarchical locking [1]. For an
>>>>>>> example, when you need to obtain the write lock for a particular
>>>>>>> Application, you do not need to lock the whole tree, but can lock only that
>>>>>>> Application itself. However, still we need to get the read locks for the
>>>>>>> parents.
>>>>>>>
>>>>>>> A separate Lock tree will be maintained for the Topology.
>>>>>>>
>>>>>>> Please share your feedback.
>>>>>>>
>>>>>>> [1].
>>>>>>> http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/2008/0801/071201hs01/071201hs01.html
>>>>>>>
>>>>>>> --
>>>>>>> Thanks and Regards,
>>>>>>>
>>>>>>> Isuru H.
>>>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Manula Chathurika Thantriwatte
>>>>>> Software Engineer
>>>>>> WSO2 Inc. : http://wso2.com
>>>>>> lean . enterprise . middleware
>>>>>>
>>>>>> email : manulac@wso2.com / manula@apache.org
>>>>>> phone : +94 772492511
>>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>>
>>>>>> --
>>>>>> <http://manulachathurika.blogspot.com/>
>>>>>> Thanks and Regards,
>>>>>>
>>>>>> Isuru H.
>>>>>> <http://manulachathurika.blogspot.com/>
>>>>>> +94 716 358 048 <http://manulachathurika.blogspot.com/>*
>>>>>> <http://wso2.com/>*
>>>>>>
>>>>>>
>>>>>> * <http://wso2.com/>*
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> --
>>>> Lahiru Sandaruwan
>>>> Committer and PMC member, Apache Stratos,
>>>> Senior Software Engineer,
>>>> WSO2 Inc., http://wso2.com
>>>> lean.enterprise.middleware
>>>>
>>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>>> blog: http://lahiruwrites.blogspot.com/
>>>> twitter: http://twitter.com/lahirus
>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>>
>>>> --
>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>>>> Thanks and Regards,
>>>>
>>>> Isuru H.
>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>>>> +94 716 358 048
>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>*
>>>> <http://wso2.com/>*
>>>>
>>>>
>>>> * <http://wso2.com/>*
>>>>
>>>>
>>>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>> --
>> Thanks and Regards,
>>
>> Isuru H.
>> +94 716 358 048* <http://wso2.com/>*
>>
>>
>> * <http://wso2.com/>*
>>
>>
>>


-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

Re: [Discuss] Hierarchical Locking for Topology

Posted by Isuru Haththotuwa <is...@apache.org>.

Thanks Imesh.

I have shown some examples for the new hierarchical locking approach.
Please do let me know your feedback on this.

   Acquire a write lock:

   From root level, acquire read lock, and acquire a write lock only for the
   relevant sub tree.

   Acquire a read lock:

   From root level, acquire read locks till the relevant sub tree

   Examples -

   Example 1: Acquiring write lock for a Cluster to modify the Cluster
object -
            acquiring:
            1. acquire read lock for all Services
            2. acquire read lock for the particular Service, to which the
cluster belongs to
            3. acquire write lock for the Cluster

            releasing:
            1. release write lock for the Cluster
            2. release read lock for the particular Service
            3. release read lock for all Services

   Example 2: Acquiring write lock to add a new Cluster object -
            acquiring:
            1. acquire read lock for all Services
            2. acquire write lock for the particular Service, to which the
cluster belongs to

            releasing:
            1. release write lock for the particular Service
            2. release read lock for all Services

   Example 3: Acquiring read lock to read Cluster information
            acquiring:
            1. acquire read lock for all Services
            2. acquire read lock for the particular Service, to which the
cluster belongs to
            3. acquire read lock for the relevant Cluster

            releasing:
            1. release read lock for the relevant Cluster
            2. release read lock for the particular Service
            3. release read lock for all Services

   Example 4: Acquiring the write lock to add a deploy a Cartridge (add a
new Service)
            acquire:
            1. acquire write lock for all Services

            release:
            1. release write lock for all Services

In all of these examples, the lock acquiring happens from top of the tree
to down. This is to avoid deadlocks.

On Mon, Oct 6, 2014 at 10:50 AM, Imesh Gunaratne <im...@apache.org> wrote:

> Great! Thanks for the clarification Isuru!
>
> Yes I agree, I think what we can do is, identify the sub trees that will
> not break the consistency of the data structure and manage locks at those
> sub tree level.
>
> Thanks
>
> On Sun, Oct 5, 2014 at 8:07 PM, Isuru Haththotuwa <is...@apache.org>
> wrote:
>
>> Hi Lahiru and Imesh,
>>
>> Thanks a lot for the input.
>>
>> What I do here is locking only the relevant sub tree of the complete
>> Topology tree, as locking the whole tree is rather inefficient. For an
>> example, when a MemberActivated event is received, we have the cluster id
>> of the cluster that particular member belongs to. IMHO, we only need to
>> acquire the write lock for that cluster , and do not need the lock for
>> complete Topology tree. Therefore, any other thread which needs to do
>> another operation on a separate sub tree (for an example, deploy a new
>> service, etc.) can do that concurrently.
>>
>> On Sun, Oct 5, 2014 at 12:20 AM, Lahiru Sandaruwan <la...@wso2.com>
>> wrote:
>>
>>> Hi Isuru,
>>>
>>> Looks like a good move to improve the efficiency,
>>>
>>> On Fri, Oct 3, 2014 at 3:05 PM, Isuru Haththotuwa <is...@apache.org>
>>> wrote:
>>>
>>>> I did the initial changes, at the testing phase now. For an example, if
>>>> we need to add a new Service, we do not need to lock everything now. We an
>>>> only acquire the write lock on Services, and add the Service.
>>>>
>>>
>>> Service can be an element of a group or an app. So shouldn't we lock
>>> complete application if we add/modify a service? Otherwise a different
>>> thread might change parents/relationships otherwise.
>>>
>> AFAIK a Service gets created when we deploy a cartridge. An
>> application/Group can refer a service. In the case of modifying an
>> Application, we do need to lock the relevant clusters that belong to that
>> Application. I implemented that.
>>
>>>
>>> So generally i think we can bring down the locking level to Application,
>>> but not the services. Also if we need to read any part, we have to get the
>>> read lock for the whole topology, such that the receiver get a particular
>>> snapshot of the topology as Imesh also mentioned.
>>>
>> If we need to lock the complete Topology, we can still do that, such as
>> in a Complete Topology event. But IMHO, if we know the exact part (a
>> particular Cluster, etc.) that we need to read/write, we do not need to
>> lock the whole Topology.
>>
>>>
>>> Thanks.
>>>
>>> If we need to modify a particular Service, can read lock the Services
>>>> and acquire the write lock on the relevant Service and do the modification.
>>>>
>>> This support is there for Services, Cluster and Applications.
>>>>
>>>
>>>> On Wed, Oct 1, 2014 at 9:27 PM, Manula Chathurika Thantriwatte <
>>>> manulac@wso2.com> wrote:
>>>>
>>>>> Hi Isuru,
>>>>>
>>>>> +1 for the hierarchical locking approach. Using hierarchical locking
>>>>> we can have more benefits [1].
>>>>>
>>>>> [1]
>>>>> http://synapticnulship.com/blog/2013/03/08/comparison-chainlocker-vs-heirarchical-mutexes/
>>>>>
>>>>>
>>>>> On Wed, Oct 1, 2014 at 7:46 PM, Isuru Haththotuwa <is...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi Devs,
>>>>>>
>>>>>> In the current Topology implementation, we acquire read/write locks
>>>>>> on Topology from the root level itself. For an example, if we need to
>>>>>> modify a single Cluster object, we still need to acquire a write lock from
>>>>>> the Topology root level. But, this is a costly operation. Specially, with
>>>>>> Service Grouping changes, we would need to traverse through an Application.
>>>>>> Since an Application can be a recursive structure, it can be a time
>>>>>> consuming operation. in such a scenario, if we are to lock the whole
>>>>>> Topology, there will be many threads waiting on that lock.
>>>>>>
>>>>>> As a solution, I think we can use hierarchical locking [1]. For an
>>>>>> example, when you need to obtain the write lock for a particular
>>>>>> Application, you do not need to lock the whole tree, but can lock only that
>>>>>> Application itself. However, still we need to get the read locks for the
>>>>>> parents.
>>>>>>
>>>>>> A separate Lock tree will be maintained for the Topology.
>>>>>>
>>>>>> Please share your feedback.
>>>>>>
>>>>>> [1].
>>>>>> http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/2008/0801/071201hs01/071201hs01.html
>>>>>>
>>>>>> --
>>>>>> Thanks and Regards,
>>>>>>
>>>>>> Isuru H.
>>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Manula Chathurika Thantriwatte
>>>>> Software Engineer
>>>>> WSO2 Inc. : http://wso2.com
>>>>> lean . enterprise . middleware
>>>>>
>>>>> email : manulac@wso2.com / manula@apache.org
>>>>> phone : +94 772492511
>>>>> blog : http://manulachathurika.blogspot.com/
>>>>>
>>>>> --
>>>>> <http://manulachathurika.blogspot.com/>
>>>>> Thanks and Regards,
>>>>>
>>>>> Isuru H.
>>>>> <http://manulachathurika.blogspot.com/>
>>>>> +94 716 358 048 <http://manulachathurika.blogspot.com/>*
>>>>> <http://wso2.com/>*
>>>>>
>>>>>
>>>>> * <http://wso2.com/>*
>>>>>
>>>>>
>>>>>
>>>
>>>
>>> --
>>> --
>>> Lahiru Sandaruwan
>>> Committer and PMC member, Apache Stratos,
>>> Senior Software Engineer,
>>> WSO2 Inc., http://wso2.com
>>> lean.enterprise.middleware
>>>
>>> email: lahirus@wso2.com cell: (+94) 773 325 954
>>> blog: http://lahiruwrites.blogspot.com/
>>> twitter: http://twitter.com/lahirus
>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>>
>>> --
>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>>> Thanks and Regards,
>>>
>>> Isuru H.
>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>>> +94 716 358 048
>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>*
>>> <http://wso2.com/>*
>>>
>>>
>>> * <http://wso2.com/>*
>>>
>>>
>>>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
> --
> Thanks and Regards,
>
> Isuru H.
> +94 716 358 048* <http://wso2.com/>*
>
>
> * <http://wso2.com/>*
>
>
>

Re: [Discuss] Hierarchical Locking for Topology

Posted by Imesh Gunaratne <im...@apache.org>.

Great! Thanks for the clarification Isuru!

Yes I agree, I think what we can do is, identify the sub trees that will
not break the consistency of the data structure and manage locks at those
sub tree level.

Thanks

On Sun, Oct 5, 2014 at 8:07 PM, Isuru Haththotuwa <is...@apache.org> wrote:

> Hi Lahiru and Imesh,
>
> Thanks a lot for the input.
>
> What I do here is locking only the relevant sub tree of the complete
> Topology tree, as locking the whole tree is rather inefficient. For an
> example, when a MemberActivated event is received, we have the cluster id
> of the cluster that particular member belongs to. IMHO, we only need to
> acquire the write lock for that cluster , and do not need the lock for
> complete Topology tree. Therefore, any other thread which needs to do
> another operation on a separate sub tree (for an example, deploy a new
> service, etc.) can do that concurrently.
>
> On Sun, Oct 5, 2014 at 12:20 AM, Lahiru Sandaruwan <la...@wso2.com>
> wrote:
>
>> Hi Isuru,
>>
>> Looks like a good move to improve the efficiency,
>>
>> On Fri, Oct 3, 2014 at 3:05 PM, Isuru Haththotuwa <is...@apache.org>
>> wrote:
>>
>>> I did the initial changes, at the testing phase now. For an example, if
>>> we need to add a new Service, we do not need to lock everything now. We an
>>> only acquire the write lock on Services, and add the Service.
>>>
>>
>> Service can be an element of a group or an app. So shouldn't we lock
>> complete application if we add/modify a service? Otherwise a different
>> thread might change parents/relationships otherwise.
>>
> AFAIK a Service gets created when we deploy a cartridge. An
> application/Group can refer a service. In the case of modifying an
> Application, we do need to lock the relevant clusters that belong to that
> Application. I implemented that.
>
>>
>> So generally i think we can bring down the locking level to Application,
>> but not the services. Also if we need to read any part, we have to get the
>> read lock for the whole topology, such that the receiver get a particular
>> snapshot of the topology as Imesh also mentioned.
>>
> If we need to lock the complete Topology, we can still do that, such as in
> a Complete Topology event. But IMHO, if we know the exact part (a
> particular Cluster, etc.) that we need to read/write, we do not need to
> lock the whole Topology.
>
>>
>> Thanks.
>>
>> If we need to modify a particular Service, can read lock the Services and
>>> acquire the write lock on the relevant Service and do the modification.
>>>
>> This support is there for Services, Cluster and Applications.
>>>
>>
>>> On Wed, Oct 1, 2014 at 9:27 PM, Manula Chathurika Thantriwatte <
>>> manulac@wso2.com> wrote:
>>>
>>>> Hi Isuru,
>>>>
>>>> +1 for the hierarchical locking approach. Using hierarchical locking we
>>>> can have more benefits [1].
>>>>
>>>> [1]
>>>> http://synapticnulship.com/blog/2013/03/08/comparison-chainlocker-vs-heirarchical-mutexes/
>>>>
>>>>
>>>> On Wed, Oct 1, 2014 at 7:46 PM, Isuru Haththotuwa <is...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Devs,
>>>>>
>>>>> In the current Topology implementation, we acquire read/write locks on
>>>>> Topology from the root level itself. For an example, if we need to modify a
>>>>> single Cluster object, we still need to acquire a write lock from the
>>>>> Topology root level. But, this is a costly operation. Specially, with
>>>>> Service Grouping changes, we would need to traverse through an Application.
>>>>> Since an Application can be a recursive structure, it can be a time
>>>>> consuming operation. in such a scenario, if we are to lock the whole
>>>>> Topology, there will be many threads waiting on that lock.
>>>>>
>>>>> As a solution, I think we can use hierarchical locking [1]. For an
>>>>> example, when you need to obtain the write lock for a particular
>>>>> Application, you do not need to lock the whole tree, but can lock only that
>>>>> Application itself. However, still we need to get the read locks for the
>>>>> parents.
>>>>>
>>>>> A separate Lock tree will be maintained for the Topology.
>>>>>
>>>>> Please share your feedback.
>>>>>
>>>>> [1].
>>>>> http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/2008/0801/071201hs01/071201hs01.html
>>>>>
>>>>> --
>>>>> Thanks and Regards,
>>>>>
>>>>> Isuru H.
>>>>> +94 716 358 048* <http://wso2.com/>*
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Manula Chathurika Thantriwatte
>>>> Software Engineer
>>>> WSO2 Inc. : http://wso2.com
>>>> lean . enterprise . middleware
>>>>
>>>> email : manulac@wso2.com / manula@apache.org
>>>> phone : +94 772492511
>>>> blog : http://manulachathurika.blogspot.com/
>>>>
>>>> --
>>>> <http://manulachathurika.blogspot.com/>
>>>> Thanks and Regards,
>>>>
>>>> Isuru H.
>>>> <http://manulachathurika.blogspot.com/>
>>>> +94 716 358 048 <http://manulachathurika.blogspot.com/>*
>>>> <http://wso2.com/>*
>>>>
>>>>
>>>> * <http://wso2.com/>*
>>>>
>>>>
>>>>
>>
>>
>> --
>> --
>> Lahiru Sandaruwan
>> Committer and PMC member, Apache Stratos,
>> Senior Software Engineer,
>> WSO2 Inc., http://wso2.com
>> lean.enterprise.middleware
>>
>> email: lahirus@wso2.com cell: (+94) 773 325 954
>> blog: http://lahiruwrites.blogspot.com/
>> twitter: http://twitter.com/lahirus
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>> --
>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>> Thanks and Regards,
>>
>> Isuru H.
>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
>> +94 716 358 048 <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>*
>> <http://wso2.com/>*
>>
>>
>> * <http://wso2.com/>*
>>
>>
>>


-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

Re: [Discuss] Hierarchical Locking for Topology

Posted by Isuru Haththotuwa <is...@apache.org>.

Hi Lahiru and Imesh,

Thanks a lot for the input.

What I do here is locking only the relevant sub tree of the complete
Topology tree, as locking the whole tree is rather inefficient. For an
example, when a MemberActivated event is received, we have the cluster id
of the cluster that particular member belongs to. IMHO, we only need to
acquire the write lock for that cluster , and do not need the lock for
complete Topology tree. Therefore, any other thread which needs to do
another operation on a separate sub tree (for an example, deploy a new
service, etc.) can do that concurrently.

On Sun, Oct 5, 2014 at 12:20 AM, Lahiru Sandaruwan <la...@wso2.com> wrote:

> Hi Isuru,
>
> Looks like a good move to improve the efficiency,
>
> On Fri, Oct 3, 2014 at 3:05 PM, Isuru Haththotuwa <is...@apache.org>
> wrote:
>
>> I did the initial changes, at the testing phase now. For an example, if
>> we need to add a new Service, we do not need to lock everything now. We an
>> only acquire the write lock on Services, and add the Service.
>>
>
> Service can be an element of a group or an app. So shouldn't we lock
> complete application if we add/modify a service? Otherwise a different
> thread might change parents/relationships otherwise.
>
AFAIK a Service gets created when we deploy a cartridge. An
application/Group can refer a service. In the case of modifying an
Application, we do need to lock the relevant clusters that belong to that
Application. I implemented that.

>
> So generally i think we can bring down the locking level to Application,
> but not the services. Also if we need to read any part, we have to get the
> read lock for the whole topology, such that the receiver get a particular
> snapshot of the topology as Imesh also mentioned.
>
If we need to lock the complete Topology, we can still do that, such as in
a Complete Topology event. But IMHO, if we know the exact part (a
particular Cluster, etc.) that we need to read/write, we do not need to
lock the whole Topology.

>
> Thanks.
>
> If we need to modify a particular Service, can read lock the Services and
>> acquire the write lock on the relevant Service and do the modification.
>>
> This support is there for Services, Cluster and Applications.
>>
>
>> On Wed, Oct 1, 2014 at 9:27 PM, Manula Chathurika Thantriwatte <
>> manulac@wso2.com> wrote:
>>
>>> Hi Isuru,
>>>
>>> +1 for the hierarchical locking approach. Using hierarchical locking we
>>> can have more benefits [1].
>>>
>>> [1]
>>> http://synapticnulship.com/blog/2013/03/08/comparison-chainlocker-vs-heirarchical-mutexes/
>>>
>>>
>>> On Wed, Oct 1, 2014 at 7:46 PM, Isuru Haththotuwa <is...@apache.org>
>>> wrote:
>>>
>>>> Hi Devs,
>>>>
>>>> In the current Topology implementation, we acquire read/write locks on
>>>> Topology from the root level itself. For an example, if we need to modify a
>>>> single Cluster object, we still need to acquire a write lock from the
>>>> Topology root level. But, this is a costly operation. Specially, with
>>>> Service Grouping changes, we would need to traverse through an Application.
>>>> Since an Application can be a recursive structure, it can be a time
>>>> consuming operation. in such a scenario, if we are to lock the whole
>>>> Topology, there will be many threads waiting on that lock.
>>>>
>>>> As a solution, I think we can use hierarchical locking [1]. For an
>>>> example, when you need to obtain the write lock for a particular
>>>> Application, you do not need to lock the whole tree, but can lock only that
>>>> Application itself. However, still we need to get the read locks for the
>>>> parents.
>>>>
>>>> A separate Lock tree will be maintained for the Topology.
>>>>
>>>> Please share your feedback.
>>>>
>>>> [1].
>>>> http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/2008/0801/071201hs01/071201hs01.html
>>>>
>>>> --
>>>> Thanks and Regards,
>>>>
>>>> Isuru H.
>>>> +94 716 358 048* <http://wso2.com/>*
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Manula Chathurika Thantriwatte
>>> Software Engineer
>>> WSO2 Inc. : http://wso2.com
>>> lean . enterprise . middleware
>>>
>>> email : manulac@wso2.com / manula@apache.org
>>> phone : +94 772492511
>>> blog : http://manulachathurika.blogspot.com/
>>>
>>> --
>>> <http://manulachathurika.blogspot.com/>
>>> Thanks and Regards,
>>>
>>> Isuru H.
>>> <http://manulachathurika.blogspot.com/>
>>> +94 716 358 048 <http://manulachathurika.blogspot.com/>*
>>> <http://wso2.com/>*
>>>
>>>
>>> * <http://wso2.com/>*
>>>
>>>
>>>
>
>
> --
> --
> Lahiru Sandaruwan
> Committer and PMC member, Apache Stratos,
> Senior Software Engineer,
> WSO2 Inc., http://wso2.com
> lean.enterprise.middleware
>
> email: lahirus@wso2.com cell: (+94) 773 325 954
> blog: http://lahiruwrites.blogspot.com/
> twitter: http://twitter.com/lahirus
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
> --
> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
> Thanks and Regards,
>
> Isuru H.
> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>
> +94 716 358 048 <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>*
> <http://wso2.com/>*
>
>
> * <http://wso2.com/>*
>
>
>

Re: [Discuss] Hierarchical Locking for Topology

Posted by Lahiru Sandaruwan <la...@wso2.com>.

Hi Isuru,

Looks like a good move to improve the efficiency,

On Fri, Oct 3, 2014 at 3:05 PM, Isuru Haththotuwa <is...@apache.org> wrote:

> I did the initial changes, at the testing phase now. For an example, if we
> need to add a new Service, we do not need to lock everything now. We an
> only acquire the write lock on Services, and add the Service.
>

Service can be an element of a group or an app. So shouldn't we lock
complete application if we add/modify a service? Otherwise a different
thread might change parents/relationships otherwise.

So generally i think we can bring down the locking level to Application,
but not the services. Also if we need to read any part, we have to get the
read lock for the whole topology, such that the receiver get a particular
snapshot of the topology as Imesh also mentioned.

Thanks.

If we need to modify a particular Service, can read lock the Services and
> acquire the write lock on the relevant Service and do the modification.
>
This support is there for Services, Cluster and Applications.
>

> On Wed, Oct 1, 2014 at 9:27 PM, Manula Chathurika Thantriwatte <
> manulac@wso2.com> wrote:
>
>> Hi Isuru,
>>
>> +1 for the hierarchical locking approach. Using hierarchical locking we
>> can have more benefits [1].
>>
>> [1]
>> http://synapticnulship.com/blog/2013/03/08/comparison-chainlocker-vs-heirarchical-mutexes/
>>
>>
>> On Wed, Oct 1, 2014 at 7:46 PM, Isuru Haththotuwa <is...@apache.org>
>> wrote:
>>
>>> Hi Devs,
>>>
>>> In the current Topology implementation, we acquire read/write locks on
>>> Topology from the root level itself. For an example, if we need to modify a
>>> single Cluster object, we still need to acquire a write lock from the
>>> Topology root level. But, this is a costly operation. Specially, with
>>> Service Grouping changes, we would need to traverse through an Application.
>>> Since an Application can be a recursive structure, it can be a time
>>> consuming operation. in such a scenario, if we are to lock the whole
>>> Topology, there will be many threads waiting on that lock.
>>>
>>> As a solution, I think we can use hierarchical locking [1]. For an
>>> example, when you need to obtain the write lock for a particular
>>> Application, you do not need to lock the whole tree, but can lock only that
>>> Application itself. However, still we need to get the read locks for the
>>> parents.
>>>
>>> A separate Lock tree will be maintained for the Topology.
>>>
>>> Please share your feedback.
>>>
>>> [1].
>>> http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/2008/0801/071201hs01/071201hs01.html
>>>
>>> --
>>> Thanks and Regards,
>>>
>>> Isuru H.
>>> +94 716 358 048* <http://wso2.com/>*
>>>
>>>
>>>
>>
>>
>> --
>> Regards,
>> Manula Chathurika Thantriwatte
>> Software Engineer
>> WSO2 Inc. : http://wso2.com
>> lean . enterprise . middleware
>>
>> email : manulac@wso2.com / manula@apache.org
>> phone : +94 772492511
>> blog : http://manulachathurika.blogspot.com/
>>
>> --
>> <http://manulachathurika.blogspot.com/>
>> Thanks and Regards,
>>
>> Isuru H.
>> <http://manulachathurika.blogspot.com/>
>> +94 716 358 048 <http://manulachathurika.blogspot.com/>*
>> <http://wso2.com/>*
>>
>>
>> * <http://wso2.com/>*
>>
>>
>>


-- 
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

email: lahirus@wso2.com cell: (+94) 773 325 954
blog: http://lahiruwrites.blogspot.com/
twitter: http://twitter.com/lahirus
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146

Re: [Discuss] Hierarchical Locking for Topology

Posted by Isuru Haththotuwa <is...@apache.org>.

I did the initial changes, at the testing phase now. For an example, if we
need to add a new Service, we do not need to lock everything now. We an
only acquire the write lock on Services, and add the Service. If we need to
modify a particular Service, can read lock the Services and acquire the
write lock on the relevant Service and do the modification. This support is
there for Services, Cluster and Applications.

On Wed, Oct 1, 2014 at 9:27 PM, Manula Chathurika Thantriwatte <
manulac@wso2.com> wrote:

> Hi Isuru,
>
> +1 for the hierarchical locking approach. Using hierarchical locking we
> can have more benefits [1].
>
> [1]
> http://synapticnulship.com/blog/2013/03/08/comparison-chainlocker-vs-heirarchical-mutexes/
>
>
> On Wed, Oct 1, 2014 at 7:46 PM, Isuru Haththotuwa <is...@apache.org>
> wrote:
>
>> Hi Devs,
>>
>> In the current Topology implementation, we acquire read/write locks on
>> Topology from the root level itself. For an example, if we need to modify a
>> single Cluster object, we still need to acquire a write lock from the
>> Topology root level. But, this is a costly operation. Specially, with
>> Service Grouping changes, we would need to traverse through an Application.
>> Since an Application can be a recursive structure, it can be a time
>> consuming operation. in such a scenario, if we are to lock the whole
>> Topology, there will be many threads waiting on that lock.
>>
>> As a solution, I think we can use hierarchical locking [1]. For an
>> example, when you need to obtain the write lock for a particular
>> Application, you do not need to lock the whole tree, but can lock only that
>> Application itself. However, still we need to get the read locks for the
>> parents.
>>
>> A separate Lock tree will be maintained for the Topology.
>>
>> Please share your feedback.
>>
>> [1].
>> http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/2008/0801/071201hs01/071201hs01.html
>>
>> --
>> Thanks and Regards,
>>
>> Isuru H.
>> +94 716 358 048* <http://wso2.com/>*
>>
>>
>>
>
>
> --
> Regards,
> Manula Chathurika Thantriwatte
> Software Engineer
> WSO2 Inc. : http://wso2.com
> lean . enterprise . middleware
>
> email : manulac@wso2.com / manula@apache.org
> phone : +94 772492511
> blog : http://manulachathurika.blogspot.com/
>
> --
> Thanks and Regards,
>
> Isuru H.
> +94 716 358 048* <http://wso2.com/>*
>
>
> * <http://wso2.com/>*
>
>
>

Re: [Discuss] Hierarchical Locking for Topology

Posted by Manula Chathurika Thantriwatte <ma...@wso2.com>.

Hi Isuru,

+1 for the hierarchical locking approach. Using hierarchical locking we can
have more benefits [1].

[1]
http://synapticnulship.com/blog/2013/03/08/comparison-chainlocker-vs-heirarchical-mutexes/


On Wed, Oct 1, 2014 at 7:46 PM, Isuru Haththotuwa <is...@apache.org> wrote:

> Hi Devs,
>
> In the current Topology implementation, we acquire read/write locks on
> Topology from the root level itself. For an example, if we need to modify a
> single Cluster object, we still need to acquire a write lock from the
> Topology root level. But, this is a costly operation. Specially, with
> Service Grouping changes, we would need to traverse through an Application.
> Since an Application can be a recursive structure, it can be a time
> consuming operation. in such a scenario, if we are to lock the whole
> Topology, there will be many threads waiting on that lock.
>
> As a solution, I think we can use hierarchical locking [1]. For an
> example, when you need to obtain the write lock for a particular
> Application, you do not need to lock the whole tree, but can lock only that
> Application itself. However, still we need to get the read locks for the
> parents.
>
> A separate Lock tree will be maintained for the Topology.
>
> Please share your feedback.
>
> [1].
> http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/2008/0801/071201hs01/071201hs01.html
>
> --
> Thanks and Regards,
>
> Isuru H.
> +94 716 358 048* <http://wso2.com/>*
>
>
>


-- 
Regards,
Manula Chathurika Thantriwatte
Software Engineer
WSO2 Inc. : http://wso2.com
lean . enterprise . middleware

email : manulac@wso2.com / manula@apache.org
phone : +94 772492511
blog : http://manulachathurika.blogspot.com/