You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2021/05/27 14:41:06 UTC

[GitHub] [helix] github-actions[bot] opened a new issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)

github-actions[bot] opened a new issue #1763:
URL: https://github.com/apache/helix/issues/1763


   This issue is created for tracking unstable test: testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang edited a comment on issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)

Posted by GitBox <gi...@apache.org>.
jiajunwang edited a comment on issue #1763:
URL: https://github.com/apache/helix/issues/1763#issuecomment-850690134


   It seems that the following NPE blocks the metrics update (for cleaning up) so the metrics are not removed as expected.
   
   ```
   2021-05-27T13:44:01.7121680Z 1810401 [ExternalViewComputeWorker-TestClusterStatusMonitorLifecycle0_2] ERROR org.apache.helix.controller.pipeline.AbstractAsyncBaseStage  - Failed to process DEFAULT::ExternalViewComputeStage asynchronously
   2021-05-27T13:44:01.7127825Z java.lang.NullPointerException
   2021-05-27T13:44:01.7130637Z 	at org.apache.helix.manager.zk.ZkBaseDataAccessor.set(ZkBaseDataAccessor.java:912)
   2021-05-27T13:44:01.7135009Z 	at org.apache.helix.manager.zk.ZkBaseDataAccessor.setChildren(ZkBaseDataAccessor.java:856)
   2021-05-27T13:44:01.7140459Z 	at org.apache.helix.manager.zk.ZKHelixDataAccessor.setChildren(ZKHelixDataAccessor.java:578)
   2021-05-27T13:44:01.7146249Z 	at org.apache.helix.controller.stages.ExternalViewComputeStage.execute(ExternalViewComputeStage.java:134)
   2021-05-27T13:44:01.7149517Z 	at org.apache.helix.controller.pipeline.AbstractAsyncBaseStage$1.run(AbstractAsyncBaseStage.java:47)
   2021-05-27T13:44:01.7152294Z 	at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:701)
   2021-05-27T13:44:01.7154919Z 	at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:697)
   2021-05-27T13:44:01.7157110Z 	at org.apache.helix.common.DedupEventProcessor.run(DedupEventProcessor.java:61)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on issue #1763:
URL: https://github.com/apache/helix/issues/1763#issuecomment-850746227


   We have attempted multiple times to fix this test. Although each of the fixes improves the whole system a little bit, the problem has not been completely fixed yet.
   Here's the latest theory (new findings).
   1. The current way of closing an async stage (EV update stage) execute thread is to interrupt it. However, since the default implementation of the task swallows the interruptedException, the thread will be kept alive.
   2. From the admin thread perspective, it shuts down the async stage threads and then cleans up all the monitors. The assumption is that after the async stage threads are all closed, there won't be anyone adding new metrics. So the cleanup of the monitors should be clean.
   3. However, due to the issue in point one, an EV update stage thread may still executing. It is possible that we have a race condition when an EV update stage event is being executed while the main thread is closing the monitors. If the closing of Resource Monitor happens earlier than the EV event processing, then the EV event processing will recreate a new Resource Monitor. 
    
   This mainly causes problems in the test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang closed issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)

Posted by GitBox <gi...@apache.org>.
jiajunwang closed issue #1763:
URL: https://github.com/apache/helix/issues/1763


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang closed issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)

Posted by GitBox <gi...@apache.org>.
jiajunwang closed issue #1763:
URL: https://github.com/apache/helix/issues/1763


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on issue #1763:
URL: https://github.com/apache/helix/issues/1763#issuecomment-850622378


   There was a bug in CI script so the link to test run was not attached.
   
   https://github.com/apache/helix/actions/runs/882175732


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] github-actions[bot] commented on issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #1763:
URL: https://github.com/apache/helix/issues/1763#issuecomment-877727031


   This test fails in: https://github.com/apache/helix/actions/runs/1018970531


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang closed issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)

Posted by GitBox <gi...@apache.org>.
jiajunwang closed issue #1763:
URL: https://github.com/apache/helix/issues/1763


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on issue #1763:
URL: https://github.com/apache/helix/issues/1763#issuecomment-850697592


   The remaining ResourceMonitor is in TestClusterStatusMonitorLifecycle0_2. I suspect the problem is related to the following error log.
   
   ```
   2021-05-27T13:44:01.7064179Z 1810401 [ExternalViewComputeWorker-TestClusterStatusMonitorLifecycle0_2] ERROR org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks  - Interrupted waiting for success
   2021-05-27T13:44:01.7069212Z java.lang.InterruptedException
   2021-05-27T13:44:01.7070305Z 	at java.lang.Object.wait(Native Method)
   2021-05-27T13:44:01.7071251Z 	at java.lang.Object.wait(Object.java:502)
   2021-05-27T13:44:01.7073554Z 	at org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks$DefaultCallback.waitForSuccess(ZkAsyncCallbacks.java:249)
   2021-05-27T13:44:01.7077636Z 	at org.apache.helix.manager.zk.ZkBaseDataAccessor.set(ZkBaseDataAccessor.java:910)
   2021-05-27T13:44:01.7082400Z 	at org.apache.helix.manager.zk.ZkBaseDataAccessor.setChildren(ZkBaseDataAccessor.java:856)
   2021-05-27T13:44:01.7087450Z 	at org.apache.helix.manager.zk.ZKHelixDataAccessor.setChildren(ZKHelixDataAccessor.java:578)
   2021-05-27T13:44:01.7093176Z 	at org.apache.helix.controller.stages.ExternalViewComputeStage.execute(ExternalViewComputeStage.java:134)
   2021-05-27T13:44:01.7099273Z 	at org.apache.helix.controller.pipeline.AbstractAsyncBaseStage$1.run(AbstractAsyncBaseStage.java:47)
   2021-05-27T13:44:01.7104274Z 	at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:701)
   2021-05-27T13:44:01.7108823Z 	at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:697)
   2021-05-27T13:44:01.7113039Z 	at org.apache.helix.common.DedupEventProcessor.run(DedupEventProcessor.java:61)
   2021-05-27T13:44:01.7121680Z 1810401 [ExternalViewComputeWorker-TestClusterStatusMonitorLifecycle0_2] ERROR org.apache.helix.controller.pipeline.AbstractAsyncBaseStage  - Failed to process DEFAULT::ExternalViewComputeStage asynchronously
   2021-05-27T13:44:01.7127825Z java.lang.NullPointerException
   2021-05-27T13:44:01.7130637Z 	at org.apache.helix.manager.zk.ZkBaseDataAccessor.set(ZkBaseDataAccessor.java:912)
   2021-05-27T13:44:01.7135009Z 	at org.apache.helix.manager.zk.ZkBaseDataAccessor.setChildren(ZkBaseDataAccessor.java:856)
   2021-05-27T13:44:01.7140459Z 	at org.apache.helix.manager.zk.ZKHelixDataAccessor.setChildren(ZKHelixDataAccessor.java:578)
   2021-05-27T13:44:01.7146249Z 	at org.apache.helix.controller.stages.ExternalViewComputeStage.execute(ExternalViewComputeStage.java:134)
   2021-05-27T13:44:01.7149517Z 	at org.apache.helix.controller.pipeline.AbstractAsyncBaseStage$1.run(AbstractAsyncBaseStage.java:47)
   2021-05-27T13:44:01.7152294Z 	at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:701)
   2021-05-27T13:44:01.7154919Z 	at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:697)
   2021-05-27T13:44:01.7157110Z 	at org.apache.helix.common.DedupEventProcessor.run(DedupEventProcessor.java:61)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on issue #1763:
URL: https://github.com/apache/helix/issues/1763#issuecomment-850690134


   It seems that the following NPE blocks the metrics update (for cleaning up) so the metrics are not removed as expected.
   
   2021-05-27T13:44:01.7121680Z 1810401 [ExternalViewComputeWorker-TestClusterStatusMonitorLifecycle0_2] ERROR org.apache.helix.controller.pipeline.AbstractAsyncBaseStage  - Failed to process DEFAULT::ExternalViewComputeStage asynchronously
   2021-05-27T13:44:01.7127825Z java.lang.NullPointerException
   2021-05-27T13:44:01.7130637Z 	at org.apache.helix.manager.zk.ZkBaseDataAccessor.set(ZkBaseDataAccessor.java:912)
   2021-05-27T13:44:01.7135009Z 	at org.apache.helix.manager.zk.ZkBaseDataAccessor.setChildren(ZkBaseDataAccessor.java:856)
   2021-05-27T13:44:01.7140459Z 	at org.apache.helix.manager.zk.ZKHelixDataAccessor.setChildren(ZKHelixDataAccessor.java:578)
   2021-05-27T13:44:01.7146249Z 	at org.apache.helix.controller.stages.ExternalViewComputeStage.execute(ExternalViewComputeStage.java:134)
   2021-05-27T13:44:01.7149517Z 	at org.apache.helix.controller.pipeline.AbstractAsyncBaseStage$1.run(AbstractAsyncBaseStage.java:47)
   2021-05-27T13:44:01.7152294Z 	at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:701)
   2021-05-27T13:44:01.7154919Z 	at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:697)
   2021-05-27T13:44:01.7157110Z 	at org.apache.helix.common.DedupEventProcessor.run(DedupEventProcessor.java:61)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] github-actions[bot] commented on issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #1763:
URL: https://github.com/apache/helix/issues/1763#issuecomment-859197826


   This test fails in: https://github.com/apache/helix/actions/runs/926889399


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang removed a comment on issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)

Posted by GitBox <gi...@apache.org>.
jiajunwang removed a comment on issue #1763:
URL: https://github.com/apache/helix/issues/1763#issuecomment-850690134


   It seems that the following NPE blocks the metrics update (for cleaning up) so the metrics are not removed as expected.
   
   ```
   2021-05-27T13:44:01.7121680Z 1810401 [ExternalViewComputeWorker-TestClusterStatusMonitorLifecycle0_2] ERROR org.apache.helix.controller.pipeline.AbstractAsyncBaseStage  - Failed to process DEFAULT::ExternalViewComputeStage asynchronously
   2021-05-27T13:44:01.7127825Z java.lang.NullPointerException
   2021-05-27T13:44:01.7130637Z 	at org.apache.helix.manager.zk.ZkBaseDataAccessor.set(ZkBaseDataAccessor.java:912)
   2021-05-27T13:44:01.7135009Z 	at org.apache.helix.manager.zk.ZkBaseDataAccessor.setChildren(ZkBaseDataAccessor.java:856)
   2021-05-27T13:44:01.7140459Z 	at org.apache.helix.manager.zk.ZKHelixDataAccessor.setChildren(ZKHelixDataAccessor.java:578)
   2021-05-27T13:44:01.7146249Z 	at org.apache.helix.controller.stages.ExternalViewComputeStage.execute(ExternalViewComputeStage.java:134)
   2021-05-27T13:44:01.7149517Z 	at org.apache.helix.controller.pipeline.AbstractAsyncBaseStage$1.run(AbstractAsyncBaseStage.java:47)
   2021-05-27T13:44:01.7152294Z 	at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:701)
   2021-05-27T13:44:01.7154919Z 	at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:697)
   2021-05-27T13:44:01.7157110Z 	at org.apache.helix.common.DedupEventProcessor.run(DedupEventProcessor.java:61)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org