PCA live test failing: MessageBrokerHATestCase

[Akila Ravihansa Perera <ra...@wso2.com>]

Hi,

It seems MessageBrokerHATestCase is failing when running the complete PCA
live test suite. But it worked when running separately. The test case was
not waiting for agent activation which might have given unexpected results.
Therefore I refactored the test case to assert the agent activation first
before testing MB fail-over scenarios.

Here is the new flow now;

 - Subscriber fail-over test case is run at first (given a high priority)
 - Assert agent activation
 - Stop the default MB, assert whether PCA detects the failure
 - Stop the second MB, ditto
 - Stop the third MB, ditto

 - Publisher fail-over test case is run
 - Assert agent activation
 - Publish InstanceCleanupMemberEvent which will trigger the PCA to publish
maintenance mode event
 - Stop the default MB as soon as InstanceCleanupMemberEvent is received by
PCA (test case will check the PCA logs)
  * There is a small window that PCA might publish maintenance mode before
MB is stopped but I couldn't figure out a better way.
 - Assert whether PCA detects the failure when publishing the maintenance
event


The test case is failing when asserting publish fail-over scenario. PCA
does not detect the MB failure when publishing an event.
@Chamila: I think there is a bug in publish fail-over logic. Will you be
able to have a look?


There is a basic flaw in the test cases, IMO. It does not assert whether
PCA is actually able to receive events via secondary MBs it is subscribed
to. The test case only asserts whether PCA detects the failure. Also
similarly in publish fail-over test case, it does not assert whether
published events to the secondary MB can be actually received via any
receivers. I think we need to assert those in order to 100% guarantee that
fail-over feature is working properly.

I made few improvements to PCA test logging as well. Now the test suite
will log messages in test cases to file:
test-integration/target/pca-live-test.log. With this we can easily see logs
coming from the test methods and logs coming from PCA. All logs are printed
to the console as before.


Thanks.


-- 
Akila Ravihansa Perera
WSO2 Inc.;  http://wso2.com/

Blog: http://ravihansa3000.blogspot.com