You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by Akila Ravihansa Perera <ra...@wso2.com> on 2015/11/29 20:25:58 UTC

Refactoring Mock IaaS, Metadata API, Messaging Processor Chain, Integration Utils

Hi,

Currently there are lot of Thread.sleep calls in mock iaas component which
makes it slow and cause unexpected behavior due to concurrency issues. Also
it has a significant performance overhead when running integration tests
since mock iaas is being used for test cases. I've been working on
improving this component by doing following changes;

 - Remove *all* Thread sleep calls in mock iaas
 - Introduce a method named 'initialize' to start event receivers and
publishers. This is a synchronous call which grantees that receiver and
publisher objects will be created successfully. If not it will throw an
exception and startInstance() method call in CC will fail. Earlier this
task was delegated to an executor service which made it difficult to check
whether mock instance was created successfully.
 - Create topology receiver in mock instance and listen for member
initialized and member started events. It will publish instance started and
instance activated events based on topology events received rather than
sleeping for some time interval before publishing.

After making those changes I faced multiple integration test failures. This
was mainly because integration tests relied heavily on Thread sleep calls
to assert various conditions. With these changes, the time taken for a mock
instance/app to become active came down to milliseconds, hence test cases
could not detect member status or app status correctly. Therefore I had to
introduce a new non-blocking mechanism to check app/member status by using
thread synchronization.

Now the average time taken for complete integration tests is around 16 mins
(earlier it was more than 30 mins). This is almost 50% performance gain.
Created JIRA at [1].


Following is a summary of changes.

AutoscalerTopologyEventReceiver:
 - Fix formatting and log messages
 - ClusterInstanceTerminatedEventListener check appMonitor is null when
calling destroy() on monitor

AutoscalerServiceImpl
 - Fix formatting and log messages

ClusterStatusActiveProcessor
 - Fix formatting and log messages (log cluster-instance-id)

GroupStatusActiveProcessor
  - Fix formatting and log messages (log group-instance-id)
  - Print groups map entries and cluster data holder map entries if debug
enabled

GroupStatusProcessor
  - Fix formatting and log messages (log group-instance-id)

GroupStatusTerminatedProcessor
  - Fix formatting and log messages (log group-instance-id)

CloudControllerServiceComponent
  - Increase THREAD_POOL_SIZE from 10 to 20


TopologyBuilder
  - Fix formatting and log messages
  - Move acquire topology lock call outside of try block

RestClient
 - Add logs for every method to help troubleshooting integration failures

StratosTestServerManager
 - Call waitForPort method with restart timeout of 600000 ms. This is to
avoid test failures due to slow builder machines.


RestConstants
 - Add entity name: REPO_NOTIFY_NAME = "GitHook"

TopologyHandler
 - Update timeout values:
APPLICATION_ACTIVATION_TIMEOUT = 300000;
APPLICATION_INACTIVATION_TIMEOUT = 120000;
APPLICATION_UNDEPLOYMENT_TIMEOUT = 30000;
MEMBER_TERMINATION_TIMEOUT = 120000;
APPLICATION_TOPOLOGY_INIT_TIMEOUT = 20000;

 - Increase executorService pool size from 10 to 30 to compensate for
additional event receivers
 - Event receivers with logs for events
     - healthStatEventReceiver: MemberFaultEvent

     - applicationsEventReceiver:
ApplicationInstanceActivatedEventListener,
ApplicationInstanceInactivatedEventListener

     - topologyEventReceiver: MemberActivatedEventListener,
MemberTerminatedEventListener, ClusterInstanceActivatedEventListener,
ClusterInstanceInactivateEventListener

 - Added logs for every method to help troubleshoot integration failures
 - Asynchronous mechanism to assertApplicationActiveStatus
 - Asynchronous mechanism to assertApplicationInactiveStatus


Application
 - Added log in getStatus() method to print status of all application
instances


ApplicationCreatedMessageProcessor
  - Fix formatting and log messages

ApplicationDeletedMessageProcessor
  - Fix formatting and log messages
ApplicationInstanceActivatedMessageProcessor
  - Fix formatting and log messages (log app-instance-id)

ApplicationInstanceCreatedMessageProcessor
 - Fix formatting and log messages (log app-instance-id)

ApplicationInstanceInactivatedMessageProcessor
- Fix formatting and log messages (log app-instance-id)

ApplicationInstanceTerminatedMessageProcessor
 - Fix formatting and log messages (log app-instance-id)

ApplicationInstanceTerminatingMessageProcessor
 - Fix formatting and log messages (log app-instance-id)

ClusterStatusClusterActivatedMessageProcessor
 - Fix formatting and log messages (log cluster-instance-id)

ClusterStatusClusterInactivateMessageProcessor
 - Fix formatting and log messages (log cluster-instance-id)

ClusterStatusClusterInstanceCreatedMessageProcessor
 - Fix formatting and log messages (log cluster-instance-id)

ClusterStatusClusterResetMessageProcessor
 - Fix formatting and log messages (log cluster-instance-id)

ClusterStatusClusterTerminatedMessageProcessor
 - Fix formatting and log messages (log cluster-instance-id)

ClusterStatusClusterTerminatingMessageProcessor
 - Fix formatting and log messages (log cluster-instance-id)

Introduce removeEventListener method for message processor chain to remove
a registered event listener object. This is needed since integration tests
will be registering event listeners on-demand. Those listeners needs to be
removed at the end of the test case.
 - ApplicationSignUpMessageProcessorChain
 - ApplicationsMessageProcessorChain
 - ClusterStatusMessageProcessorChain
 - DomainMappingMessageProcessorChain
 - HealthStatMessageProcessorChain
 - InitializerMessageProcessorChain
 - InstanceNotifierMessageProcessorChain
 - InstanceStatusMessageProcessorChain
 - TenantMessageProcessorChain
 - TopologyMessageProcessorChain
 - MessageProcessorChain
 - ApplicationsEventMessageDelegator
 - ApplicationsEventReceiver

InstanceNotifierEventReceiver
 - synchronized blocks for execute() and terminate()
 - eventSubscriber object creation moved to constructor from execute()
method. This is to avoid possible NPE when calling terminate() method

MetadataApi
 - Catch generic Exception instead of RegistryException

DataStore
 - throws MetadataException in addition to RegistryException
MetadataApiRegistry
 - throw new MetadataException instead of RegistryException


MockIaasServiceComponent
 - Move mockIaasServiceUtil.startInstancesPersisted() to
MockIaasServiceImpl()

MockIaasServiceImpl
 - start persisted mock instances in the constructor
 - Remove all thread sleep calls


MockIaasServiceUtil
 - Move startInstancesPersisted() method to MockIaasServiceImpl()


MockInstance
 - Create TopologyEventReceiver object and register:
MemberInitializedEventListener, MemberStartedEventListener,
MemberMaintenanceListener
 - Publish InstanceStartedEvent upon receiving MemberInitializedEvent
 - Publish InstanceActivatedEvent upon receiving MemberStartedEvent
 - Publish InstanceReadyToShutdownEvent upon receiving
MemberMaintenanceModeEvent
 - synchronized terminate() and initialize() methods
 - Introduce an initialize() method to start event receivers and health
stat publisher
 - Introduce field 'memberStatus' of type MemberStatus to track the
life-cycle of mock instance

MockIaasServiceTest
 - Start embedded MB in unit test case since mockIaasService.startInstance
call will start the event receivers as well

GitHookTestCase
 - Make artifactUpdateEventCount an AtomicInteger
 - Replace restClient.doPost call with restClient.addEntity call
 - Terminate instanceNotifierEventReceiver at the end of test case
 - Shutdown eventListenerExecutorService at the end of test case

[1] https://issues.apache.org/jira/browse/STRATOS-1633

Thanks.

-- 
Akila Ravihansa Perera
WSO2 Inc.;  http://wso2.com/

Blog: http://ravihansa3000.blogspot.com

Re: Refactoring Mock IaaS, Metadata API, Messaging Processor Chain, Integration Utils

Posted by Rajkumar Rajaratnam <ra...@wso2.com>.
Awesome improvements Akila. Mock is super fast now :)

Thanks,
Raj.

On Mon, Nov 30, 2015 at 1:33 PM, Gayan Gunarathne <ga...@wso2.com> wrote:

> Great work Akila. It is really nice to see that it will minimize the time
> taken to run the integration tests.
>
> Thanks,
> Gayan
>
> On Mon, Nov 30, 2015 at 12:58 PM, Reka Thirunavukkarasu <re...@wso2.com>
> wrote:
>
>> Hi Akila
>>
>> It is really great work in identifying and optimizing lot of places in
>> the source code. We didn't review the complete flow earlier and improve any
>> of the time. This effort will definitely improve the overall performance
>> and reduce any intermittent concurrent issues that we faced with
>> integration tests sometimes. Nice to see that 20+ integration test is
>> running in 15min.
>>
>> Thanks,
>> Reka
>>
>> On Mon, Nov 30, 2015 at 12:55 AM, Akila Ravihansa Perera <
>> ravihansa@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> Currently there are lot of Thread.sleep calls in mock iaas component
>>> which makes it slow and cause unexpected behavior due to concurrency
>>> issues. Also it has a significant performance overhead when running
>>> integration tests since mock iaas is being used for test cases. I've been
>>> working on improving this component by doing following changes;
>>>
>>>  - Remove *all* Thread sleep calls in mock iaas
>>>  - Introduce a method named 'initialize' to start event receivers and
>>> publishers. This is a synchronous call which grantees that receiver and
>>> publisher objects will be created successfully. If not it will throw an
>>> exception and startInstance() method call in CC will fail. Earlier this
>>> task was delegated to an executor service which made it difficult to check
>>> whether mock instance was created successfully.
>>>  - Create topology receiver in mock instance and listen for member
>>> initialized and member started events. It will publish instance started and
>>> instance activated events based on topology events received rather than
>>> sleeping for some time interval before publishing.
>>>
>>> After making those changes I faced multiple integration test failures.
>>> This was mainly because integration tests relied heavily on Thread sleep
>>> calls to assert various conditions. With these changes, the time taken for
>>> a mock instance/app to become active came down to milliseconds, hence test
>>> cases could not detect member status or app status correctly. Therefore I
>>> had to introduce a new non-blocking mechanism to check app/member status by
>>> using thread synchronization.
>>>
>>> Now the average time taken for complete integration tests is around 16
>>> mins (earlier it was more than 30 mins). This is almost 50% performance
>>> gain. Created JIRA at [1].
>>>
>>>
>>> Following is a summary of changes.
>>>
>>> AutoscalerTopologyEventReceiver:
>>>  - Fix formatting and log messages
>>>  - ClusterInstanceTerminatedEventListener check appMonitor is null when
>>> calling destroy() on monitor
>>>
>>> AutoscalerServiceImpl
>>>  - Fix formatting and log messages
>>>
>>> ClusterStatusActiveProcessor
>>>  - Fix formatting and log messages (log cluster-instance-id)
>>>
>>> GroupStatusActiveProcessor
>>>   - Fix formatting and log messages (log group-instance-id)
>>>   - Print groups map entries and cluster data holder map entries if
>>> debug enabled
>>>
>>> GroupStatusProcessor
>>>   - Fix formatting and log messages (log group-instance-id)
>>>
>>> GroupStatusTerminatedProcessor
>>>   - Fix formatting and log messages (log group-instance-id)
>>>
>>> CloudControllerServiceComponent
>>>   - Increase THREAD_POOL_SIZE from 10 to 20
>>>
>>>
>>> TopologyBuilder
>>>   - Fix formatting and log messages
>>>   - Move acquire topology lock call outside of try block
>>>
>>> RestClient
>>>  - Add logs for every method to help troubleshooting integration failures
>>>
>>> StratosTestServerManager
>>>  - Call waitForPort method with restart timeout of 600000 ms. This is to
>>> avoid test failures due to slow builder machines.
>>>
>>>
>>> RestConstants
>>>  - Add entity name: REPO_NOTIFY_NAME = "GitHook"
>>>
>>> TopologyHandler
>>>  - Update timeout values:
>>> APPLICATION_ACTIVATION_TIMEOUT = 300000;
>>> APPLICATION_INACTIVATION_TIMEOUT = 120000;
>>> APPLICATION_UNDEPLOYMENT_TIMEOUT = 30000;
>>> MEMBER_TERMINATION_TIMEOUT = 120000;
>>> APPLICATION_TOPOLOGY_INIT_TIMEOUT = 20000;
>>>
>>>  - Increase executorService pool size from 10 to 30 to compensate for
>>> additional event receivers
>>>  - Event receivers with logs for events
>>>      - healthStatEventReceiver: MemberFaultEvent
>>>
>>>      - applicationsEventReceiver:
>>> ApplicationInstanceActivatedEventListener,
>>> ApplicationInstanceInactivatedEventListener
>>>
>>>      - topologyEventReceiver: MemberActivatedEventListener,
>>> MemberTerminatedEventListener, ClusterInstanceActivatedEventListener,
>>> ClusterInstanceInactivateEventListener
>>>
>>>  - Added logs for every method to help troubleshoot integration failures
>>>  - Asynchronous mechanism to assertApplicationActiveStatus
>>>  - Asynchronous mechanism to assertApplicationInactiveStatus
>>>
>>>
>>> Application
>>>  - Added log in getStatus() method to print status of all application
>>> instances
>>>
>>>
>>> ApplicationCreatedMessageProcessor
>>>   - Fix formatting and log messages
>>>
>>> ApplicationDeletedMessageProcessor
>>>   - Fix formatting and log messages
>>> ApplicationInstanceActivatedMessageProcessor
>>>   - Fix formatting and log messages (log app-instance-id)
>>>
>>> ApplicationInstanceCreatedMessageProcessor
>>>  - Fix formatting and log messages (log app-instance-id)
>>>
>>> ApplicationInstanceInactivatedMessageProcessor
>>> - Fix formatting and log messages (log app-instance-id)
>>>
>>> ApplicationInstanceTerminatedMessageProcessor
>>>  - Fix formatting and log messages (log app-instance-id)
>>>
>>> ApplicationInstanceTerminatingMessageProcessor
>>>  - Fix formatting and log messages (log app-instance-id)
>>>
>>> ClusterStatusClusterActivatedMessageProcessor
>>>  - Fix formatting and log messages (log cluster-instance-id)
>>>
>>> ClusterStatusClusterInactivateMessageProcessor
>>>  - Fix formatting and log messages (log cluster-instance-id)
>>>
>>> ClusterStatusClusterInstanceCreatedMessageProcessor
>>>  - Fix formatting and log messages (log cluster-instance-id)
>>>
>>> ClusterStatusClusterResetMessageProcessor
>>>  - Fix formatting and log messages (log cluster-instance-id)
>>>
>>> ClusterStatusClusterTerminatedMessageProcessor
>>>  - Fix formatting and log messages (log cluster-instance-id)
>>>
>>> ClusterStatusClusterTerminatingMessageProcessor
>>>  - Fix formatting and log messages (log cluster-instance-id)
>>>
>>> Introduce removeEventListener method for message processor chain to
>>> remove a registered event listener object. This is needed since integration
>>> tests will be registering event listeners on-demand. Those listeners needs
>>> to be removed at the end of the test case.
>>>  - ApplicationSignUpMessageProcessorChain
>>>  - ApplicationsMessageProcessorChain
>>>  - ClusterStatusMessageProcessorChain
>>>  - DomainMappingMessageProcessorChain
>>>  - HealthStatMessageProcessorChain
>>>  - InitializerMessageProcessorChain
>>>  - InstanceNotifierMessageProcessorChain
>>>  - InstanceStatusMessageProcessorChain
>>>  - TenantMessageProcessorChain
>>>  - TopologyMessageProcessorChain
>>>  - MessageProcessorChain
>>>  - ApplicationsEventMessageDelegator
>>>  - ApplicationsEventReceiver
>>>
>>> InstanceNotifierEventReceiver
>>>  - synchronized blocks for execute() and terminate()
>>>  - eventSubscriber object creation moved to constructor from execute()
>>> method. This is to avoid possible NPE when calling terminate() method
>>>
>>> MetadataApi
>>>  - Catch generic Exception instead of RegistryException
>>>
>>> DataStore
>>>  - throws MetadataException in addition to RegistryException
>>> MetadataApiRegistry
>>>  - throw new MetadataException instead of RegistryException
>>>
>>>
>>> MockIaasServiceComponent
>>>  - Move mockIaasServiceUtil.startInstancesPersisted() to
>>> MockIaasServiceImpl()
>>>
>>> MockIaasServiceImpl
>>>  - start persisted mock instances in the constructor
>>>  - Remove all thread sleep calls
>>>
>>>
>>> MockIaasServiceUtil
>>>  - Move startInstancesPersisted() method to MockIaasServiceImpl()
>>>
>>>
>>> MockInstance
>>>  - Create TopologyEventReceiver object and register:
>>> MemberInitializedEventListener, MemberStartedEventListener,
>>> MemberMaintenanceListener
>>>  - Publish InstanceStartedEvent upon receiving MemberInitializedEvent
>>>  - Publish InstanceActivatedEvent upon receiving MemberStartedEvent
>>>  - Publish InstanceReadyToShutdownEvent upon receiving
>>> MemberMaintenanceModeEvent
>>>  - synchronized terminate() and initialize() methods
>>>  - Introduce an initialize() method to start event receivers and health
>>> stat publisher
>>>  - Introduce field 'memberStatus' of type MemberStatus to track the
>>> life-cycle of mock instance
>>>
>>> MockIaasServiceTest
>>>  - Start embedded MB in unit test case since
>>> mockIaasService.startInstance call will start the event receivers as well
>>>
>>> GitHookTestCase
>>>  - Make artifactUpdateEventCount an AtomicInteger
>>>  - Replace restClient.doPost call with restClient.addEntity call
>>>  - Terminate instanceNotifierEventReceiver at the end of test case
>>>  - Shutdown eventListenerExecutorService at the end of test case
>>>
>>> [1] https://issues.apache.org/jira/browse/STRATOS-1633
>>>
>>> Thanks.
>>>
>>> --
>>> Akila Ravihansa Perera
>>> WSO2 Inc.;  http://wso2.com/
>>>
>>> Blog: http://ravihansa3000.blogspot.com
>>>
>>
>>
>>
>> --
>> Reka Thirunavukkarasu
>> Senior Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>> Mobile: +94776442007
>>
>>
>>
>
>
> --
>
> Gayan Gunarathne
> Technical Lead, WSO2 Inc. (http://wso2.com)
> Committer & PMC Member, Apache Stratos
> email : gayang@wso2.com  | mobile : +94 775030545 <%2B94%20766819985>
>
>
>



-- 
Rajkumar Rajaratnam
Committer & PMC Member, Apache Stratos
Software Engineer, WSO2

Mobile : +94777568639
Blog : rajkumarr.com

Re: Refactoring Mock IaaS, Metadata API, Messaging Processor Chain, Integration Utils

Posted by Gayan Gunarathne <ga...@wso2.com>.
Great work Akila. It is really nice to see that it will minimize the time
taken to run the integration tests.

Thanks,
Gayan

On Mon, Nov 30, 2015 at 12:58 PM, Reka Thirunavukkarasu <re...@wso2.com>
wrote:

> Hi Akila
>
> It is really great work in identifying and optimizing lot of places in the
> source code. We didn't review the complete flow earlier and improve any of
> the time. This effort will definitely improve the overall performance and
> reduce any intermittent concurrent issues that we faced with integration
> tests sometimes. Nice to see that 20+ integration test is running in 15min.
>
> Thanks,
> Reka
>
> On Mon, Nov 30, 2015 at 12:55 AM, Akila Ravihansa Perera <
> ravihansa@wso2.com> wrote:
>
>> Hi,
>>
>> Currently there are lot of Thread.sleep calls in mock iaas component
>> which makes it slow and cause unexpected behavior due to concurrency
>> issues. Also it has a significant performance overhead when running
>> integration tests since mock iaas is being used for test cases. I've been
>> working on improving this component by doing following changes;
>>
>>  - Remove *all* Thread sleep calls in mock iaas
>>  - Introduce a method named 'initialize' to start event receivers and
>> publishers. This is a synchronous call which grantees that receiver and
>> publisher objects will be created successfully. If not it will throw an
>> exception and startInstance() method call in CC will fail. Earlier this
>> task was delegated to an executor service which made it difficult to check
>> whether mock instance was created successfully.
>>  - Create topology receiver in mock instance and listen for member
>> initialized and member started events. It will publish instance started and
>> instance activated events based on topology events received rather than
>> sleeping for some time interval before publishing.
>>
>> After making those changes I faced multiple integration test failures.
>> This was mainly because integration tests relied heavily on Thread sleep
>> calls to assert various conditions. With these changes, the time taken for
>> a mock instance/app to become active came down to milliseconds, hence test
>> cases could not detect member status or app status correctly. Therefore I
>> had to introduce a new non-blocking mechanism to check app/member status by
>> using thread synchronization.
>>
>> Now the average time taken for complete integration tests is around 16
>> mins (earlier it was more than 30 mins). This is almost 50% performance
>> gain. Created JIRA at [1].
>>
>>
>> Following is a summary of changes.
>>
>> AutoscalerTopologyEventReceiver:
>>  - Fix formatting and log messages
>>  - ClusterInstanceTerminatedEventListener check appMonitor is null when
>> calling destroy() on monitor
>>
>> AutoscalerServiceImpl
>>  - Fix formatting and log messages
>>
>> ClusterStatusActiveProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> GroupStatusActiveProcessor
>>   - Fix formatting and log messages (log group-instance-id)
>>   - Print groups map entries and cluster data holder map entries if debug
>> enabled
>>
>> GroupStatusProcessor
>>   - Fix formatting and log messages (log group-instance-id)
>>
>> GroupStatusTerminatedProcessor
>>   - Fix formatting and log messages (log group-instance-id)
>>
>> CloudControllerServiceComponent
>>   - Increase THREAD_POOL_SIZE from 10 to 20
>>
>>
>> TopologyBuilder
>>   - Fix formatting and log messages
>>   - Move acquire topology lock call outside of try block
>>
>> RestClient
>>  - Add logs for every method to help troubleshooting integration failures
>>
>> StratosTestServerManager
>>  - Call waitForPort method with restart timeout of 600000 ms. This is to
>> avoid test failures due to slow builder machines.
>>
>>
>> RestConstants
>>  - Add entity name: REPO_NOTIFY_NAME = "GitHook"
>>
>> TopologyHandler
>>  - Update timeout values:
>> APPLICATION_ACTIVATION_TIMEOUT = 300000;
>> APPLICATION_INACTIVATION_TIMEOUT = 120000;
>> APPLICATION_UNDEPLOYMENT_TIMEOUT = 30000;
>> MEMBER_TERMINATION_TIMEOUT = 120000;
>> APPLICATION_TOPOLOGY_INIT_TIMEOUT = 20000;
>>
>>  - Increase executorService pool size from 10 to 30 to compensate for
>> additional event receivers
>>  - Event receivers with logs for events
>>      - healthStatEventReceiver: MemberFaultEvent
>>
>>      - applicationsEventReceiver:
>> ApplicationInstanceActivatedEventListener,
>> ApplicationInstanceInactivatedEventListener
>>
>>      - topologyEventReceiver: MemberActivatedEventListener,
>> MemberTerminatedEventListener, ClusterInstanceActivatedEventListener,
>> ClusterInstanceInactivateEventListener
>>
>>  - Added logs for every method to help troubleshoot integration failures
>>  - Asynchronous mechanism to assertApplicationActiveStatus
>>  - Asynchronous mechanism to assertApplicationInactiveStatus
>>
>>
>> Application
>>  - Added log in getStatus() method to print status of all application
>> instances
>>
>>
>> ApplicationCreatedMessageProcessor
>>   - Fix formatting and log messages
>>
>> ApplicationDeletedMessageProcessor
>>   - Fix formatting and log messages
>> ApplicationInstanceActivatedMessageProcessor
>>   - Fix formatting and log messages (log app-instance-id)
>>
>> ApplicationInstanceCreatedMessageProcessor
>>  - Fix formatting and log messages (log app-instance-id)
>>
>> ApplicationInstanceInactivatedMessageProcessor
>> - Fix formatting and log messages (log app-instance-id)
>>
>> ApplicationInstanceTerminatedMessageProcessor
>>  - Fix formatting and log messages (log app-instance-id)
>>
>> ApplicationInstanceTerminatingMessageProcessor
>>  - Fix formatting and log messages (log app-instance-id)
>>
>> ClusterStatusClusterActivatedMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> ClusterStatusClusterInactivateMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> ClusterStatusClusterInstanceCreatedMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> ClusterStatusClusterResetMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> ClusterStatusClusterTerminatedMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> ClusterStatusClusterTerminatingMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> Introduce removeEventListener method for message processor chain to
>> remove a registered event listener object. This is needed since integration
>> tests will be registering event listeners on-demand. Those listeners needs
>> to be removed at the end of the test case.
>>  - ApplicationSignUpMessageProcessorChain
>>  - ApplicationsMessageProcessorChain
>>  - ClusterStatusMessageProcessorChain
>>  - DomainMappingMessageProcessorChain
>>  - HealthStatMessageProcessorChain
>>  - InitializerMessageProcessorChain
>>  - InstanceNotifierMessageProcessorChain
>>  - InstanceStatusMessageProcessorChain
>>  - TenantMessageProcessorChain
>>  - TopologyMessageProcessorChain
>>  - MessageProcessorChain
>>  - ApplicationsEventMessageDelegator
>>  - ApplicationsEventReceiver
>>
>> InstanceNotifierEventReceiver
>>  - synchronized blocks for execute() and terminate()
>>  - eventSubscriber object creation moved to constructor from execute()
>> method. This is to avoid possible NPE when calling terminate() method
>>
>> MetadataApi
>>  - Catch generic Exception instead of RegistryException
>>
>> DataStore
>>  - throws MetadataException in addition to RegistryException
>> MetadataApiRegistry
>>  - throw new MetadataException instead of RegistryException
>>
>>
>> MockIaasServiceComponent
>>  - Move mockIaasServiceUtil.startInstancesPersisted() to
>> MockIaasServiceImpl()
>>
>> MockIaasServiceImpl
>>  - start persisted mock instances in the constructor
>>  - Remove all thread sleep calls
>>
>>
>> MockIaasServiceUtil
>>  - Move startInstancesPersisted() method to MockIaasServiceImpl()
>>
>>
>> MockInstance
>>  - Create TopologyEventReceiver object and register:
>> MemberInitializedEventListener, MemberStartedEventListener,
>> MemberMaintenanceListener
>>  - Publish InstanceStartedEvent upon receiving MemberInitializedEvent
>>  - Publish InstanceActivatedEvent upon receiving MemberStartedEvent
>>  - Publish InstanceReadyToShutdownEvent upon receiving
>> MemberMaintenanceModeEvent
>>  - synchronized terminate() and initialize() methods
>>  - Introduce an initialize() method to start event receivers and health
>> stat publisher
>>  - Introduce field 'memberStatus' of type MemberStatus to track the
>> life-cycle of mock instance
>>
>> MockIaasServiceTest
>>  - Start embedded MB in unit test case since
>> mockIaasService.startInstance call will start the event receivers as well
>>
>> GitHookTestCase
>>  - Make artifactUpdateEventCount an AtomicInteger
>>  - Replace restClient.doPost call with restClient.addEntity call
>>  - Terminate instanceNotifierEventReceiver at the end of test case
>>  - Shutdown eventListenerExecutorService at the end of test case
>>
>> [1] https://issues.apache.org/jira/browse/STRATOS-1633
>>
>> Thanks.
>>
>> --
>> Akila Ravihansa Perera
>> WSO2 Inc.;  http://wso2.com/
>>
>> Blog: http://ravihansa3000.blogspot.com
>>
>
>
>
> --
> Reka Thirunavukkarasu
> Senior Software Engineer,
> WSO2, Inc.:http://wso2.com,
> Mobile: +94776442007
>
>
>


-- 

Gayan Gunarathne
Technical Lead, WSO2 Inc. (http://wso2.com)
Committer & PMC Member, Apache Stratos
email : gayang@wso2.com  | mobile : +94 775030545 <%2B94%20766819985>

Re: Refactoring Mock IaaS, Metadata API, Messaging Processor Chain, Integration Utils

Posted by Reka Thirunavukkarasu <re...@wso2.com>.
Hi Akila

It is really great work in identifying and optimizing lot of places in the
source code. We didn't review the complete flow earlier and improve any of
the time. This effort will definitely improve the overall performance and
reduce any intermittent concurrent issues that we faced with integration
tests sometimes. Nice to see that 20+ integration test is running in 15min.

Thanks,
Reka

On Mon, Nov 30, 2015 at 12:55 AM, Akila Ravihansa Perera <ravihansa@wso2.com
> wrote:

> Hi,
>
> Currently there are lot of Thread.sleep calls in mock iaas component which
> makes it slow and cause unexpected behavior due to concurrency issues. Also
> it has a significant performance overhead when running integration tests
> since mock iaas is being used for test cases. I've been working on
> improving this component by doing following changes;
>
>  - Remove *all* Thread sleep calls in mock iaas
>  - Introduce a method named 'initialize' to start event receivers and
> publishers. This is a synchronous call which grantees that receiver and
> publisher objects will be created successfully. If not it will throw an
> exception and startInstance() method call in CC will fail. Earlier this
> task was delegated to an executor service which made it difficult to check
> whether mock instance was created successfully.
>  - Create topology receiver in mock instance and listen for member
> initialized and member started events. It will publish instance started and
> instance activated events based on topology events received rather than
> sleeping for some time interval before publishing.
>
> After making those changes I faced multiple integration test failures.
> This was mainly because integration tests relied heavily on Thread sleep
> calls to assert various conditions. With these changes, the time taken for
> a mock instance/app to become active came down to milliseconds, hence test
> cases could not detect member status or app status correctly. Therefore I
> had to introduce a new non-blocking mechanism to check app/member status by
> using thread synchronization.
>
> Now the average time taken for complete integration tests is around 16
> mins (earlier it was more than 30 mins). This is almost 50% performance
> gain. Created JIRA at [1].
>
>
> Following is a summary of changes.
>
> AutoscalerTopologyEventReceiver:
>  - Fix formatting and log messages
>  - ClusterInstanceTerminatedEventListener check appMonitor is null when
> calling destroy() on monitor
>
> AutoscalerServiceImpl
>  - Fix formatting and log messages
>
> ClusterStatusActiveProcessor
>  - Fix formatting and log messages (log cluster-instance-id)
>
> GroupStatusActiveProcessor
>   - Fix formatting and log messages (log group-instance-id)
>   - Print groups map entries and cluster data holder map entries if debug
> enabled
>
> GroupStatusProcessor
>   - Fix formatting and log messages (log group-instance-id)
>
> GroupStatusTerminatedProcessor
>   - Fix formatting and log messages (log group-instance-id)
>
> CloudControllerServiceComponent
>   - Increase THREAD_POOL_SIZE from 10 to 20
>
>
> TopologyBuilder
>   - Fix formatting and log messages
>   - Move acquire topology lock call outside of try block
>
> RestClient
>  - Add logs for every method to help troubleshooting integration failures
>
> StratosTestServerManager
>  - Call waitForPort method with restart timeout of 600000 ms. This is to
> avoid test failures due to slow builder machines.
>
>
> RestConstants
>  - Add entity name: REPO_NOTIFY_NAME = "GitHook"
>
> TopologyHandler
>  - Update timeout values:
> APPLICATION_ACTIVATION_TIMEOUT = 300000;
> APPLICATION_INACTIVATION_TIMEOUT = 120000;
> APPLICATION_UNDEPLOYMENT_TIMEOUT = 30000;
> MEMBER_TERMINATION_TIMEOUT = 120000;
> APPLICATION_TOPOLOGY_INIT_TIMEOUT = 20000;
>
>  - Increase executorService pool size from 10 to 30 to compensate for
> additional event receivers
>  - Event receivers with logs for events
>      - healthStatEventReceiver: MemberFaultEvent
>
>      - applicationsEventReceiver:
> ApplicationInstanceActivatedEventListener,
> ApplicationInstanceInactivatedEventListener
>
>      - topologyEventReceiver: MemberActivatedEventListener,
> MemberTerminatedEventListener, ClusterInstanceActivatedEventListener,
> ClusterInstanceInactivateEventListener
>
>  - Added logs for every method to help troubleshoot integration failures
>  - Asynchronous mechanism to assertApplicationActiveStatus
>  - Asynchronous mechanism to assertApplicationInactiveStatus
>
>
> Application
>  - Added log in getStatus() method to print status of all application
> instances
>
>
> ApplicationCreatedMessageProcessor
>   - Fix formatting and log messages
>
> ApplicationDeletedMessageProcessor
>   - Fix formatting and log messages
> ApplicationInstanceActivatedMessageProcessor
>   - Fix formatting and log messages (log app-instance-id)
>
> ApplicationInstanceCreatedMessageProcessor
>  - Fix formatting and log messages (log app-instance-id)
>
> ApplicationInstanceInactivatedMessageProcessor
> - Fix formatting and log messages (log app-instance-id)
>
> ApplicationInstanceTerminatedMessageProcessor
>  - Fix formatting and log messages (log app-instance-id)
>
> ApplicationInstanceTerminatingMessageProcessor
>  - Fix formatting and log messages (log app-instance-id)
>
> ClusterStatusClusterActivatedMessageProcessor
>  - Fix formatting and log messages (log cluster-instance-id)
>
> ClusterStatusClusterInactivateMessageProcessor
>  - Fix formatting and log messages (log cluster-instance-id)
>
> ClusterStatusClusterInstanceCreatedMessageProcessor
>  - Fix formatting and log messages (log cluster-instance-id)
>
> ClusterStatusClusterResetMessageProcessor
>  - Fix formatting and log messages (log cluster-instance-id)
>
> ClusterStatusClusterTerminatedMessageProcessor
>  - Fix formatting and log messages (log cluster-instance-id)
>
> ClusterStatusClusterTerminatingMessageProcessor
>  - Fix formatting and log messages (log cluster-instance-id)
>
> Introduce removeEventListener method for message processor chain to remove
> a registered event listener object. This is needed since integration tests
> will be registering event listeners on-demand. Those listeners needs to be
> removed at the end of the test case.
>  - ApplicationSignUpMessageProcessorChain
>  - ApplicationsMessageProcessorChain
>  - ClusterStatusMessageProcessorChain
>  - DomainMappingMessageProcessorChain
>  - HealthStatMessageProcessorChain
>  - InitializerMessageProcessorChain
>  - InstanceNotifierMessageProcessorChain
>  - InstanceStatusMessageProcessorChain
>  - TenantMessageProcessorChain
>  - TopologyMessageProcessorChain
>  - MessageProcessorChain
>  - ApplicationsEventMessageDelegator
>  - ApplicationsEventReceiver
>
> InstanceNotifierEventReceiver
>  - synchronized blocks for execute() and terminate()
>  - eventSubscriber object creation moved to constructor from execute()
> method. This is to avoid possible NPE when calling terminate() method
>
> MetadataApi
>  - Catch generic Exception instead of RegistryException
>
> DataStore
>  - throws MetadataException in addition to RegistryException
> MetadataApiRegistry
>  - throw new MetadataException instead of RegistryException
>
>
> MockIaasServiceComponent
>  - Move mockIaasServiceUtil.startInstancesPersisted() to
> MockIaasServiceImpl()
>
> MockIaasServiceImpl
>  - start persisted mock instances in the constructor
>  - Remove all thread sleep calls
>
>
> MockIaasServiceUtil
>  - Move startInstancesPersisted() method to MockIaasServiceImpl()
>
>
> MockInstance
>  - Create TopologyEventReceiver object and register:
> MemberInitializedEventListener, MemberStartedEventListener,
> MemberMaintenanceListener
>  - Publish InstanceStartedEvent upon receiving MemberInitializedEvent
>  - Publish InstanceActivatedEvent upon receiving MemberStartedEvent
>  - Publish InstanceReadyToShutdownEvent upon receiving
> MemberMaintenanceModeEvent
>  - synchronized terminate() and initialize() methods
>  - Introduce an initialize() method to start event receivers and health
> stat publisher
>  - Introduce field 'memberStatus' of type MemberStatus to track the
> life-cycle of mock instance
>
> MockIaasServiceTest
>  - Start embedded MB in unit test case since mockIaasService.startInstance
> call will start the event receivers as well
>
> GitHookTestCase
>  - Make artifactUpdateEventCount an AtomicInteger
>  - Replace restClient.doPost call with restClient.addEntity call
>  - Terminate instanceNotifierEventReceiver at the end of test case
>  - Shutdown eventListenerExecutorService at the end of test case
>
> [1] https://issues.apache.org/jira/browse/STRATOS-1633
>
> Thanks.
>
> --
> Akila Ravihansa Perera
> WSO2 Inc.;  http://wso2.com/
>
> Blog: http://ravihansa3000.blogspot.com
>



-- 
Reka Thirunavukkarasu
Senior Software Engineer,
WSO2, Inc.:http://wso2.com,
Mobile: +94776442007

Re: Refactoring Mock IaaS, Metadata API, Messaging Processor Chain, Integration Utils

Posted by Isuru Haththotuwa <is...@apache.org>.
Superb work Akila! Integration test are about three times faster now.
Earlier it used to take ~45 mins, now run in ~15 mins.

On Wed, Dec 2, 2015 at 6:48 PM, Imesh Gunaratne <im...@apache.org> wrote:

> Fabulous work Akila!! :-)
>
> On Mon, Nov 30, 2015 at 12:55 AM, Akila Ravihansa Perera <
> ravihansa@wso2.com> wrote:
>
>> Hi,
>>
>> Currently there are lot of Thread.sleep calls in mock iaas component
>> which makes it slow and cause unexpected behavior due to concurrency
>> issues. Also it has a significant performance overhead when running
>> integration tests since mock iaas is being used for test cases. I've been
>> working on improving this component by doing following changes;
>>
>>  - Remove *all* Thread sleep calls in mock iaas
>>  - Introduce a method named 'initialize' to start event receivers and
>> publishers. This is a synchronous call which grantees that receiver and
>> publisher objects will be created successfully. If not it will throw an
>> exception and startInstance() method call in CC will fail. Earlier this
>> task was delegated to an executor service which made it difficult to check
>> whether mock instance was created successfully.
>>  - Create topology receiver in mock instance and listen for member
>> initialized and member started events. It will publish instance started and
>> instance activated events based on topology events received rather than
>> sleeping for some time interval before publishing.
>>
>> After making those changes I faced multiple integration test failures.
>> This was mainly because integration tests relied heavily on Thread sleep
>> calls to assert various conditions. With these changes, the time taken for
>> a mock instance/app to become active came down to milliseconds, hence test
>> cases could not detect member status or app status correctly. Therefore I
>> had to introduce a new non-blocking mechanism to check app/member status by
>> using thread synchronization.
>>
>> Now the average time taken for complete integration tests is around 16
>> mins (earlier it was more than 30 mins). This is almost 50% performance
>> gain. Created JIRA at [1].
>>
>>
>> Following is a summary of changes.
>>
>> AutoscalerTopologyEventReceiver:
>>  - Fix formatting and log messages
>>  - ClusterInstanceTerminatedEventListener check appMonitor is null when
>> calling destroy() on monitor
>>
>> AutoscalerServiceImpl
>>  - Fix formatting and log messages
>>
>> ClusterStatusActiveProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> GroupStatusActiveProcessor
>>   - Fix formatting and log messages (log group-instance-id)
>>   - Print groups map entries and cluster data holder map entries if debug
>> enabled
>>
>> GroupStatusProcessor
>>   - Fix formatting and log messages (log group-instance-id)
>>
>> GroupStatusTerminatedProcessor
>>   - Fix formatting and log messages (log group-instance-id)
>>
>> CloudControllerServiceComponent
>>   - Increase THREAD_POOL_SIZE from 10 to 20
>>
>>
>> TopologyBuilder
>>   - Fix formatting and log messages
>>   - Move acquire topology lock call outside of try block
>>
>> RestClient
>>  - Add logs for every method to help troubleshooting integration failures
>>
>> StratosTestServerManager
>>  - Call waitForPort method with restart timeout of 600000 ms. This is to
>> avoid test failures due to slow builder machines.
>>
>>
>> RestConstants
>>  - Add entity name: REPO_NOTIFY_NAME = "GitHook"
>>
>> TopologyHandler
>>  - Update timeout values:
>> APPLICATION_ACTIVATION_TIMEOUT = 300000;
>> APPLICATION_INACTIVATION_TIMEOUT = 120000;
>> APPLICATION_UNDEPLOYMENT_TIMEOUT = 30000;
>> MEMBER_TERMINATION_TIMEOUT = 120000;
>> APPLICATION_TOPOLOGY_INIT_TIMEOUT = 20000;
>>
>>  - Increase executorService pool size from 10 to 30 to compensate for
>> additional event receivers
>>  - Event receivers with logs for events
>>      - healthStatEventReceiver: MemberFaultEvent
>>
>>      - applicationsEventReceiver:
>> ApplicationInstanceActivatedEventListener,
>> ApplicationInstanceInactivatedEventListener
>>
>>      - topologyEventReceiver: MemberActivatedEventListener,
>> MemberTerminatedEventListener, ClusterInstanceActivatedEventListener,
>> ClusterInstanceInactivateEventListener
>>
>>  - Added logs for every method to help troubleshoot integration failures
>>  - Asynchronous mechanism to assertApplicationActiveStatus
>>  - Asynchronous mechanism to assertApplicationInactiveStatus
>>
>>
>> Application
>>  - Added log in getStatus() method to print status of all application
>> instances
>>
>>
>> ApplicationCreatedMessageProcessor
>>   - Fix formatting and log messages
>>
>> ApplicationDeletedMessageProcessor
>>   - Fix formatting and log messages
>> ApplicationInstanceActivatedMessageProcessor
>>   - Fix formatting and log messages (log app-instance-id)
>>
>> ApplicationInstanceCreatedMessageProcessor
>>  - Fix formatting and log messages (log app-instance-id)
>>
>> ApplicationInstanceInactivatedMessageProcessor
>> - Fix formatting and log messages (log app-instance-id)
>>
>> ApplicationInstanceTerminatedMessageProcessor
>>  - Fix formatting and log messages (log app-instance-id)
>>
>> ApplicationInstanceTerminatingMessageProcessor
>>  - Fix formatting and log messages (log app-instance-id)
>>
>> ClusterStatusClusterActivatedMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> ClusterStatusClusterInactivateMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> ClusterStatusClusterInstanceCreatedMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> ClusterStatusClusterResetMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> ClusterStatusClusterTerminatedMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> ClusterStatusClusterTerminatingMessageProcessor
>>  - Fix formatting and log messages (log cluster-instance-id)
>>
>> Introduce removeEventListener method for message processor chain to
>> remove a registered event listener object. This is needed since integration
>> tests will be registering event listeners on-demand. Those listeners needs
>> to be removed at the end of the test case.
>>  - ApplicationSignUpMessageProcessorChain
>>  - ApplicationsMessageProcessorChain
>>  - ClusterStatusMessageProcessorChain
>>  - DomainMappingMessageProcessorChain
>>  - HealthStatMessageProcessorChain
>>  - InitializerMessageProcessorChain
>>  - InstanceNotifierMessageProcessorChain
>>  - InstanceStatusMessageProcessorChain
>>  - TenantMessageProcessorChain
>>  - TopologyMessageProcessorChain
>>  - MessageProcessorChain
>>  - ApplicationsEventMessageDelegator
>>  - ApplicationsEventReceiver
>>
>> InstanceNotifierEventReceiver
>>  - synchronized blocks for execute() and terminate()
>>  - eventSubscriber object creation moved to constructor from execute()
>> method. This is to avoid possible NPE when calling terminate() method
>>
>> MetadataApi
>>  - Catch generic Exception instead of RegistryException
>>
>> DataStore
>>  - throws MetadataException in addition to RegistryException
>> MetadataApiRegistry
>>  - throw new MetadataException instead of RegistryException
>>
>>
>> MockIaasServiceComponent
>>  - Move mockIaasServiceUtil.startInstancesPersisted() to
>> MockIaasServiceImpl()
>>
>> MockIaasServiceImpl
>>  - start persisted mock instances in the constructor
>>  - Remove all thread sleep calls
>>
>>
>> MockIaasServiceUtil
>>  - Move startInstancesPersisted() method to MockIaasServiceImpl()
>>
>>
>> MockInstance
>>  - Create TopologyEventReceiver object and register:
>> MemberInitializedEventListener, MemberStartedEventListener,
>> MemberMaintenanceListener
>>  - Publish InstanceStartedEvent upon receiving MemberInitializedEvent
>>  - Publish InstanceActivatedEvent upon receiving MemberStartedEvent
>>  - Publish InstanceReadyToShutdownEvent upon receiving
>> MemberMaintenanceModeEvent
>>  - synchronized terminate() and initialize() methods
>>  - Introduce an initialize() method to start event receivers and health
>> stat publisher
>>  - Introduce field 'memberStatus' of type MemberStatus to track the
>> life-cycle of mock instance
>>
>> MockIaasServiceTest
>>  - Start embedded MB in unit test case since
>> mockIaasService.startInstance call will start the event receivers as well
>>
>> GitHookTestCase
>>  - Make artifactUpdateEventCount an AtomicInteger
>>  - Replace restClient.doPost call with restClient.addEntity call
>>  - Terminate instanceNotifierEventReceiver at the end of test case
>>  - Shutdown eventListenerExecutorService at the end of test case
>>
>> [1] https://issues.apache.org/jira/browse/STRATOS-1633
>>
>> Thanks.
>>
>> --
>> Akila Ravihansa Perera
>> WSO2 Inc.;  http://wso2.com/
>>
>> Blog: http://ravihansa3000.blogspot.com
>>
>
>
>
> --
> Imesh Gunaratne
>
> Senior Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>
> --
> Thanks and Regards,
>
> Isuru H.
> +94 716 358 048* <http://wso2.com/>*
>
>
> * <http://wso2.com/>*
>
>
>

Re: Refactoring Mock IaaS, Metadata API, Messaging Processor Chain, Integration Utils

Posted by Imesh Gunaratne <im...@apache.org>.
Fabulous work Akila!! :-)

On Mon, Nov 30, 2015 at 12:55 AM, Akila Ravihansa Perera <ravihansa@wso2.com
> wrote:

> Hi,
>
> Currently there are lot of Thread.sleep calls in mock iaas component which
> makes it slow and cause unexpected behavior due to concurrency issues. Also
> it has a significant performance overhead when running integration tests
> since mock iaas is being used for test cases. I've been working on
> improving this component by doing following changes;
>
>  - Remove *all* Thread sleep calls in mock iaas
>  - Introduce a method named 'initialize' to start event receivers and
> publishers. This is a synchronous call which grantees that receiver and
> publisher objects will be created successfully. If not it will throw an
> exception and startInstance() method call in CC will fail. Earlier this
> task was delegated to an executor service which made it difficult to check
> whether mock instance was created successfully.
>  - Create topology receiver in mock instance and listen for member
> initialized and member started events. It will publish instance started and
> instance activated events based on topology events received rather than
> sleeping for some time interval before publishing.
>
> After making those changes I faced multiple integration test failures.
> This was mainly because integration tests relied heavily on Thread sleep
> calls to assert various conditions. With these changes, the time taken for
> a mock instance/app to become active came down to milliseconds, hence test
> cases could not detect member status or app status correctly. Therefore I
> had to introduce a new non-blocking mechanism to check app/member status by
> using thread synchronization.
>
> Now the average time taken for complete integration tests is around 16
> mins (earlier it was more than 30 mins). This is almost 50% performance
> gain. Created JIRA at [1].
>
>
> Following is a summary of changes.
>
> AutoscalerTopologyEventReceiver:
>  - Fix formatting and log messages
>  - ClusterInstanceTerminatedEventListener check appMonitor is null when
> calling destroy() on monitor
>
> AutoscalerServiceImpl
>  - Fix formatting and log messages
>
> ClusterStatusActiveProcessor
>  - Fix formatting and log messages (log cluster-instance-id)
>
> GroupStatusActiveProcessor
>   - Fix formatting and log messages (log group-instance-id)
>   - Print groups map entries and cluster data holder map entries if debug
> enabled
>
> GroupStatusProcessor
>   - Fix formatting and log messages (log group-instance-id)
>
> GroupStatusTerminatedProcessor
>   - Fix formatting and log messages (log group-instance-id)
>
> CloudControllerServiceComponent
>   - Increase THREAD_POOL_SIZE from 10 to 20
>
>
> TopologyBuilder
>   - Fix formatting and log messages
>   - Move acquire topology lock call outside of try block
>
> RestClient
>  - Add logs for every method to help troubleshooting integration failures
>
> StratosTestServerManager
>  - Call waitForPort method with restart timeout of 600000 ms. This is to
> avoid test failures due to slow builder machines.
>
>
> RestConstants
>  - Add entity name: REPO_NOTIFY_NAME = "GitHook"
>
> TopologyHandler
>  - Update timeout values:
> APPLICATION_ACTIVATION_TIMEOUT = 300000;
> APPLICATION_INACTIVATION_TIMEOUT = 120000;
> APPLICATION_UNDEPLOYMENT_TIMEOUT = 30000;
> MEMBER_TERMINATION_TIMEOUT = 120000;
> APPLICATION_TOPOLOGY_INIT_TIMEOUT = 20000;
>
>  - Increase executorService pool size from 10 to 30 to compensate for
> additional event receivers
>  - Event receivers with logs for events
>      - healthStatEventReceiver: MemberFaultEvent
>
>      - applicationsEventReceiver:
> ApplicationInstanceActivatedEventListener,
> ApplicationInstanceInactivatedEventListener
>
>      - topologyEventReceiver: MemberActivatedEventListener,
> MemberTerminatedEventListener, ClusterInstanceActivatedEventListener,
> ClusterInstanceInactivateEventListener
>
>  - Added logs for every method to help troubleshoot integration failures
>  - Asynchronous mechanism to assertApplicationActiveStatus
>  - Asynchronous mechanism to assertApplicationInactiveStatus
>
>
> Application
>  - Added log in getStatus() method to print status of all application
> instances
>
>
> ApplicationCreatedMessageProcessor
>   - Fix formatting and log messages
>
> ApplicationDeletedMessageProcessor
>   - Fix formatting and log messages
> ApplicationInstanceActivatedMessageProcessor
>   - Fix formatting and log messages (log app-instance-id)
>
> ApplicationInstanceCreatedMessageProcessor
>  - Fix formatting and log messages (log app-instance-id)
>
> ApplicationInstanceInactivatedMessageProcessor
> - Fix formatting and log messages (log app-instance-id)
>
> ApplicationInstanceTerminatedMessageProcessor
>  - Fix formatting and log messages (log app-instance-id)
>
> ApplicationInstanceTerminatingMessageProcessor
>  - Fix formatting and log messages (log app-instance-id)
>
> ClusterStatusClusterActivatedMessageProcessor
>  - Fix formatting and log messages (log cluster-instance-id)
>
> ClusterStatusClusterInactivateMessageProcessor
>  - Fix formatting and log messages (log cluster-instance-id)
>
> ClusterStatusClusterInstanceCreatedMessageProcessor
>  - Fix formatting and log messages (log cluster-instance-id)
>
> ClusterStatusClusterResetMessageProcessor
>  - Fix formatting and log messages (log cluster-instance-id)
>
> ClusterStatusClusterTerminatedMessageProcessor
>  - Fix formatting and log messages (log cluster-instance-id)
>
> ClusterStatusClusterTerminatingMessageProcessor
>  - Fix formatting and log messages (log cluster-instance-id)
>
> Introduce removeEventListener method for message processor chain to remove
> a registered event listener object. This is needed since integration tests
> will be registering event listeners on-demand. Those listeners needs to be
> removed at the end of the test case.
>  - ApplicationSignUpMessageProcessorChain
>  - ApplicationsMessageProcessorChain
>  - ClusterStatusMessageProcessorChain
>  - DomainMappingMessageProcessorChain
>  - HealthStatMessageProcessorChain
>  - InitializerMessageProcessorChain
>  - InstanceNotifierMessageProcessorChain
>  - InstanceStatusMessageProcessorChain
>  - TenantMessageProcessorChain
>  - TopologyMessageProcessorChain
>  - MessageProcessorChain
>  - ApplicationsEventMessageDelegator
>  - ApplicationsEventReceiver
>
> InstanceNotifierEventReceiver
>  - synchronized blocks for execute() and terminate()
>  - eventSubscriber object creation moved to constructor from execute()
> method. This is to avoid possible NPE when calling terminate() method
>
> MetadataApi
>  - Catch generic Exception instead of RegistryException
>
> DataStore
>  - throws MetadataException in addition to RegistryException
> MetadataApiRegistry
>  - throw new MetadataException instead of RegistryException
>
>
> MockIaasServiceComponent
>  - Move mockIaasServiceUtil.startInstancesPersisted() to
> MockIaasServiceImpl()
>
> MockIaasServiceImpl
>  - start persisted mock instances in the constructor
>  - Remove all thread sleep calls
>
>
> MockIaasServiceUtil
>  - Move startInstancesPersisted() method to MockIaasServiceImpl()
>
>
> MockInstance
>  - Create TopologyEventReceiver object and register:
> MemberInitializedEventListener, MemberStartedEventListener,
> MemberMaintenanceListener
>  - Publish InstanceStartedEvent upon receiving MemberInitializedEvent
>  - Publish InstanceActivatedEvent upon receiving MemberStartedEvent
>  - Publish InstanceReadyToShutdownEvent upon receiving
> MemberMaintenanceModeEvent
>  - synchronized terminate() and initialize() methods
>  - Introduce an initialize() method to start event receivers and health
> stat publisher
>  - Introduce field 'memberStatus' of type MemberStatus to track the
> life-cycle of mock instance
>
> MockIaasServiceTest
>  - Start embedded MB in unit test case since mockIaasService.startInstance
> call will start the event receivers as well
>
> GitHookTestCase
>  - Make artifactUpdateEventCount an AtomicInteger
>  - Replace restClient.doPost call with restClient.addEntity call
>  - Terminate instanceNotifierEventReceiver at the end of test case
>  - Shutdown eventListenerExecutorService at the end of test case
>
> [1] https://issues.apache.org/jira/browse/STRATOS-1633
>
> Thanks.
>
> --
> Akila Ravihansa Perera
> WSO2 Inc.;  http://wso2.com/
>
> Blog: http://ravihansa3000.blogspot.com
>



-- 
Imesh Gunaratne

Senior Technical Lead, WSO2
Committer & PMC Member, Apache Stratos