Re: Refactoring Mock IaaS, Metadata API, Messaging Processor Chain, Integration Utils

[Rajkumar Rajaratnam <ra...@wso2.com>]

Awesome improvements Akila. Mock is super fast now :)

Thanks,
Raj.

On Mon, Nov 30, 2015 at 1:33 PM, Gayan Gunarathne <ga...@wso2.com> wrote:

> Great work Akila. It is really nice to see that it will minimize the time
> taken to run the integration tests.
>
> Thanks,
> Gayan
>
> On Mon, Nov 30, 2015 at 12:58 PM, Reka Thirunavukkarasu <re...@wso2.com>
> wrote:
>
>> Hi Akila
>>
>> It is really great work in identifying and optimizing lot of places in
>> the source code. We didn't review the complete flow earlier and improve any
>> of the time. This effort will definitely improve the overall performance
>> and reduce any intermittent concurrent issues that we faced with
>> integration tests sometimes. Nice to see that 20+ integration test is
>> running in 15min.
>>
>> Thanks,
>> Reka
>>
>> On Mon, Nov 30, 2015 at 12:55 AM, Akila Ravihansa Perera <
>> ravihansa@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> Currently there are lot of Thread.sleep calls in mock iaas component
>>> which makes it slow and cause unexpected behavior due to concurrency
>>> issues. Also it has a significant performance overhead when running
>>> integration tests since mock iaas is being used for test cases. I've been
>>> working on improving this component by doing following changes;
>>>
>>>  - Remove *all* Thread sleep calls in mock iaas
>>>  - Introduce a method named 'initialize' to start event receivers and
>>> publishers. This is a synchronous call which grantees that receiver and
>>> publisher objects will be created successfully. If not it will throw an
>>> exception and startInstance() method call in CC will fail. Earlier this
>>> task was delegated to an executor service which made it difficult to check
>>> whether mock instance was created successfully.
>>>  - Create topology receiver in mock instance and listen for member
>>> initialized and member started events. It will publish instance started and
>>> instance activated events based on topology events received rather than
>>> sleeping for some time interval before publishing.
>>>
>>> After making those changes I faced multiple integration test failures.
>>> This was mainly because integration tests relied heavily on Thread sleep
>>> calls to assert various conditions. With these changes, the time taken for
>>> a mock instance/app to become active came down to milliseconds, hence test
>>> cases could not detect member status or app status correctly. Therefore I
>>> had to introduce a new non-blocking mechanism to check app/member status by
>>> using thread synchronization.
>>>
>>> Now the average time taken for complete integration tests is around 16
>>> mins (earlier it was more than 30 mins). This is almost 50% performance
>>> gain. Created JIRA at [1].
>>>
>>>
>>> Following is a summary of changes.
>>>
>>> AutoscalerTopologyEventReceiver:
>>>  - Fix formatting and log messages
>>>  - ClusterInstanceTerminatedEventListener check appMonitor is null when
>>> calling destroy() on monitor
>>>
>>> AutoscalerServiceImpl
>>>  - Fix formatting and log messages
>>>
>>> ClusterStatusActiveProcessor
>>>  - Fix formatting and log messages (log cluster-instance-id)
>>>
>>> GroupStatusActiveProcessor
>>>   - Fix formatting and log messages (log group-instance-id)
>>>   - Print groups map entries and cluster data holder map entries if
>>> debug enabled
>>>
>>> GroupStatusProcessor
>>>   - Fix formatting and log messages (log group-instance-id)
>>>
>>> GroupStatusTerminatedProcessor
>>>   - Fix formatting and log messages (log group-instance-id)
>>>
>>> CloudControllerServiceComponent
>>>   - Increase THREAD_POOL_SIZE from 10 to 20
>>>
>>>
>>> TopologyBuilder
>>>   - Fix formatting and log messages
>>>   - Move acquire topology lock call outside of try block
>>>
>>> RestClient
>>>  - Add logs for every method to help troubleshooting integration failures
>>>
>>> StratosTestServerManager
>>>  - Call waitForPort method with restart timeout of 600000 ms. This is to
>>> avoid test failures due to slow builder machines.
>>>
>>>
>>> RestConstants
>>>  - Add entity name: REPO_NOTIFY_NAME = "GitHook"
>>>
>>> TopologyHandler
>>>  - Update timeout values:
>>> APPLICATION_ACTIVATION_TIMEOUT = 300000;
>>> APPLICATION_INACTIVATION_TIMEOUT = 120000;
>>> APPLICATION_UNDEPLOYMENT_TIMEOUT = 30000;
>>> MEMBER_TERMINATION_TIMEOUT = 120000;
>>> APPLICATION_TOPOLOGY_INIT_TIMEOUT = 20000;
>>>
>>>  - Increase executorService pool size from 10 to 30 to compensate for
>>> additional event receivers
>>>  - Event receivers with logs for events
>>>      - healthStatEventReceiver: MemberFaultEvent
>>>
>>>      - applicationsEventReceiver:
>>> ApplicationInstanceActivatedEventListener,
>>> ApplicationInstanceInactivatedEventListener
>>>
>>>      - topologyEventReceiver: MemberActivatedEventListener,
>>> MemberTerminatedEventListener, ClusterInstanceActivatedEventListener,
>>> ClusterInstanceInactivateEventListener
>>>
>>>  - Added logs for every method to help troubleshoot integration failures
>>>  - Asynchronous mechanism to assertApplicationActiveStatus
>>>  - Asynchronous mechanism to assertApplicationInactiveStatus
>>>
>>>
>>> Application
>>>  - Added log in getStatus() method to print status of all application
>>> instances
>>>
>>>
>>> ApplicationCreatedMessageProcessor
>>>   - Fix formatting and log messages
>>>
>>> ApplicationDeletedMessageProcessor
>>>   - Fix formatting and log messages
>>> ApplicationInstanceActivatedMessageProcessor
>>>   - Fix formatting and log messages (log app-instance-id)
>>>
>>> ApplicationInstanceCreatedMessageProcessor
>>>  - Fix formatting and log messages (log app-instance-id)
>>>
>>> ApplicationInstanceInactivatedMessageProcessor
>>> - Fix formatting and log messages (log app-instance-id)
>>>
>>> ApplicationInstanceTerminatedMessageProcessor
>>>  - Fix formatting and log messages (log app-instance-id)
>>>
>>> ApplicationInstanceTerminatingMessageProcessor
>>>  - Fix formatting and log messages (log app-instance-id)
>>>
>>> ClusterStatusClusterActivatedMessageProcessor
>>>  - Fix formatting and log messages (log cluster-instance-id)
>>>
>>> ClusterStatusClusterInactivateMessageProcessor
>>>  - Fix formatting and log messages (log cluster-instance-id)
>>>
>>> ClusterStatusClusterInstanceCreatedMessageProcessor
>>>  - Fix formatting and log messages (log cluster-instance-id)
>>>
>>> ClusterStatusClusterResetMessageProcessor
>>>  - Fix formatting and log messages (log cluster-instance-id)
>>>
>>> ClusterStatusClusterTerminatedMessageProcessor
>>>  - Fix formatting and log messages (log cluster-instance-id)
>>>
>>> ClusterStatusClusterTerminatingMessageProcessor
>>>  - Fix formatting and log messages (log cluster-instance-id)
>>>
>>> Introduce removeEventListener method for message processor chain to
>>> remove a registered event listener object. This is needed since integration
>>> tests will be registering event listeners on-demand. Those listeners needs
>>> to be removed at the end of the test case.
>>>  - ApplicationSignUpMessageProcessorChain
>>>  - ApplicationsMessageProcessorChain
>>>  - ClusterStatusMessageProcessorChain
>>>  - DomainMappingMessageProcessorChain
>>>  - HealthStatMessageProcessorChain
>>>  - InitializerMessageProcessorChain
>>>  - InstanceNotifierMessageProcessorChain
>>>  - InstanceStatusMessageProcessorChain
>>>  - TenantMessageProcessorChain
>>>  - TopologyMessageProcessorChain
>>>  - MessageProcessorChain
>>>  - ApplicationsEventMessageDelegator
>>>  - ApplicationsEventReceiver
>>>
>>> InstanceNotifierEventReceiver
>>>  - synchronized blocks for execute() and terminate()
>>>  - eventSubscriber object creation moved to constructor from execute()
>>> method. This is to avoid possible NPE when calling terminate() method
>>>
>>> MetadataApi
>>>  - Catch generic Exception instead of RegistryException
>>>
>>> DataStore
>>>  - throws MetadataException in addition to RegistryException
>>> MetadataApiRegistry
>>>  - throw new MetadataException instead of RegistryException
>>>
>>>
>>> MockIaasServiceComponent
>>>  - Move mockIaasServiceUtil.startInstancesPersisted() to
>>> MockIaasServiceImpl()
>>>
>>> MockIaasServiceImpl
>>>  - start persisted mock instances in the constructor
>>>  - Remove all thread sleep calls
>>>
>>>
>>> MockIaasServiceUtil
>>>  - Move startInstancesPersisted() method to MockIaasServiceImpl()
>>>
>>>
>>> MockInstance
>>>  - Create TopologyEventReceiver object and register:
>>> MemberInitializedEventListener, MemberStartedEventListener,
>>> MemberMaintenanceListener
>>>  - Publish InstanceStartedEvent upon receiving MemberInitializedEvent
>>>  - Publish InstanceActivatedEvent upon receiving MemberStartedEvent
>>>  - Publish InstanceReadyToShutdownEvent upon receiving
>>> MemberMaintenanceModeEvent
>>>  - synchronized terminate() and initialize() methods
>>>  - Introduce an initialize() method to start event receivers and health
>>> stat publisher
>>>  - Introduce field 'memberStatus' of type MemberStatus to track the
>>> life-cycle of mock instance
>>>
>>> MockIaasServiceTest
>>>  - Start embedded MB in unit test case since
>>> mockIaasService.startInstance call will start the event receivers as well
>>>
>>> GitHookTestCase
>>>  - Make artifactUpdateEventCount an AtomicInteger
>>>  - Replace restClient.doPost call with restClient.addEntity call
>>>  - Terminate instanceNotifierEventReceiver at the end of test case
>>>  - Shutdown eventListenerExecutorService at the end of test case
>>>
>>> [1] https://issues.apache.org/jira/browse/STRATOS-1633
>>>
>>> Thanks.
>>>
>>> --
>>> Akila Ravihansa Perera
>>> WSO2 Inc.;  http://wso2.com/
>>>
>>> Blog: http://ravihansa3000.blogspot.com
>>>
>>
>>
>>
>> --
>> Reka Thirunavukkarasu
>> Senior Software Engineer,
>> WSO2, Inc.:http://wso2.com,
>> Mobile: +94776442007
>>
>>
>>
>
>
> --
>
> Gayan Gunarathne
> Technical Lead, WSO2 Inc. (http://wso2.com)
> Committer & PMC Member, Apache Stratos
> email : gayang@wso2.com  | mobile : +94 775030545 <%2B94%20766819985>
>
>
>



-- 
Rajkumar Rajaratnam
Committer & PMC Member, Apache Stratos
Software Engineer, WSO2

Mobile : +94777568639
Blog : rajkumarr.com