You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2021/05/27 14:41:06 UTC
[GitHub] [helix] github-actions[bot] opened a new issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)
github-actions[bot] opened a new issue #1763:
URL: https://github.com/apache/helix/issues/1763
This issue is created for tracking unstable test: testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang edited a comment on issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)
Posted by GitBox <gi...@apache.org>.
jiajunwang edited a comment on issue #1763:
URL: https://github.com/apache/helix/issues/1763#issuecomment-850690134
It seems that the following NPE blocks the metrics update (for cleaning up) so the metrics are not removed as expected.
```
2021-05-27T13:44:01.7121680Z 1810401 [ExternalViewComputeWorker-TestClusterStatusMonitorLifecycle0_2] ERROR org.apache.helix.controller.pipeline.AbstractAsyncBaseStage - Failed to process DEFAULT::ExternalViewComputeStage asynchronously
2021-05-27T13:44:01.7127825Z java.lang.NullPointerException
2021-05-27T13:44:01.7130637Z at org.apache.helix.manager.zk.ZkBaseDataAccessor.set(ZkBaseDataAccessor.java:912)
2021-05-27T13:44:01.7135009Z at org.apache.helix.manager.zk.ZkBaseDataAccessor.setChildren(ZkBaseDataAccessor.java:856)
2021-05-27T13:44:01.7140459Z at org.apache.helix.manager.zk.ZKHelixDataAccessor.setChildren(ZKHelixDataAccessor.java:578)
2021-05-27T13:44:01.7146249Z at org.apache.helix.controller.stages.ExternalViewComputeStage.execute(ExternalViewComputeStage.java:134)
2021-05-27T13:44:01.7149517Z at org.apache.helix.controller.pipeline.AbstractAsyncBaseStage$1.run(AbstractAsyncBaseStage.java:47)
2021-05-27T13:44:01.7152294Z at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:701)
2021-05-27T13:44:01.7154919Z at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:697)
2021-05-27T13:44:01.7157110Z at org.apache.helix.common.DedupEventProcessor.run(DedupEventProcessor.java:61)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on issue #1763:
URL: https://github.com/apache/helix/issues/1763#issuecomment-850746227
We have attempted multiple times to fix this test. Although each of the fixes improves the whole system a little bit, the problem has not been completely fixed yet.
Here's the latest theory (new findings).
1. The current way of closing an async stage (EV update stage) execute thread is to interrupt it. However, since the default implementation of the task swallows the interruptedException, the thread will be kept alive.
2. From the admin thread perspective, it shuts down the async stage threads and then cleans up all the monitors. The assumption is that after the async stage threads are all closed, there won't be anyone adding new metrics. So the cleanup of the monitors should be clean.
3. However, due to the issue in point one, an EV update stage thread may still executing. It is possible that we have a race condition when an EV update stage event is being executed while the main thread is closing the monitors. If the closing of Resource Monitor happens earlier than the EV event processing, then the EV event processing will recreate a new Resource Monitor.
This mainly causes problems in the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang closed issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)
Posted by GitBox <gi...@apache.org>.
jiajunwang closed issue #1763:
URL: https://github.com/apache/helix/issues/1763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang closed issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)
Posted by GitBox <gi...@apache.org>.
jiajunwang closed issue #1763:
URL: https://github.com/apache/helix/issues/1763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on issue #1763:
URL: https://github.com/apache/helix/issues/1763#issuecomment-850622378
There was a bug in CI script so the link to test run was not attached.
https://github.com/apache/helix/actions/runs/882175732
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] github-actions[bot] commented on issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #1763:
URL: https://github.com/apache/helix/issues/1763#issuecomment-877727031
This test fails in: https://github.com/apache/helix/actions/runs/1018970531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang closed issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)
Posted by GitBox <gi...@apache.org>.
jiajunwang closed issue #1763:
URL: https://github.com/apache/helix/issues/1763
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on issue #1763:
URL: https://github.com/apache/helix/issues/1763#issuecomment-850697592
The remaining ResourceMonitor is in TestClusterStatusMonitorLifecycle0_2. I suspect the problem is related to the following error log.
```
2021-05-27T13:44:01.7064179Z 1810401 [ExternalViewComputeWorker-TestClusterStatusMonitorLifecycle0_2] ERROR org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks - Interrupted waiting for success
2021-05-27T13:44:01.7069212Z java.lang.InterruptedException
2021-05-27T13:44:01.7070305Z at java.lang.Object.wait(Native Method)
2021-05-27T13:44:01.7071251Z at java.lang.Object.wait(Object.java:502)
2021-05-27T13:44:01.7073554Z at org.apache.helix.zookeeper.zkclient.callback.ZkAsyncCallbacks$DefaultCallback.waitForSuccess(ZkAsyncCallbacks.java:249)
2021-05-27T13:44:01.7077636Z at org.apache.helix.manager.zk.ZkBaseDataAccessor.set(ZkBaseDataAccessor.java:910)
2021-05-27T13:44:01.7082400Z at org.apache.helix.manager.zk.ZkBaseDataAccessor.setChildren(ZkBaseDataAccessor.java:856)
2021-05-27T13:44:01.7087450Z at org.apache.helix.manager.zk.ZKHelixDataAccessor.setChildren(ZKHelixDataAccessor.java:578)
2021-05-27T13:44:01.7093176Z at org.apache.helix.controller.stages.ExternalViewComputeStage.execute(ExternalViewComputeStage.java:134)
2021-05-27T13:44:01.7099273Z at org.apache.helix.controller.pipeline.AbstractAsyncBaseStage$1.run(AbstractAsyncBaseStage.java:47)
2021-05-27T13:44:01.7104274Z at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:701)
2021-05-27T13:44:01.7108823Z at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:697)
2021-05-27T13:44:01.7113039Z at org.apache.helix.common.DedupEventProcessor.run(DedupEventProcessor.java:61)
2021-05-27T13:44:01.7121680Z 1810401 [ExternalViewComputeWorker-TestClusterStatusMonitorLifecycle0_2] ERROR org.apache.helix.controller.pipeline.AbstractAsyncBaseStage - Failed to process DEFAULT::ExternalViewComputeStage asynchronously
2021-05-27T13:44:01.7127825Z java.lang.NullPointerException
2021-05-27T13:44:01.7130637Z at org.apache.helix.manager.zk.ZkBaseDataAccessor.set(ZkBaseDataAccessor.java:912)
2021-05-27T13:44:01.7135009Z at org.apache.helix.manager.zk.ZkBaseDataAccessor.setChildren(ZkBaseDataAccessor.java:856)
2021-05-27T13:44:01.7140459Z at org.apache.helix.manager.zk.ZKHelixDataAccessor.setChildren(ZKHelixDataAccessor.java:578)
2021-05-27T13:44:01.7146249Z at org.apache.helix.controller.stages.ExternalViewComputeStage.execute(ExternalViewComputeStage.java:134)
2021-05-27T13:44:01.7149517Z at org.apache.helix.controller.pipeline.AbstractAsyncBaseStage$1.run(AbstractAsyncBaseStage.java:47)
2021-05-27T13:44:01.7152294Z at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:701)
2021-05-27T13:44:01.7154919Z at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:697)
2021-05-27T13:44:01.7157110Z at org.apache.helix.common.DedupEventProcessor.run(DedupEventProcessor.java:61)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on issue #1763:
URL: https://github.com/apache/helix/issues/1763#issuecomment-850690134
It seems that the following NPE blocks the metrics update (for cleaning up) so the metrics are not removed as expected.
2021-05-27T13:44:01.7121680Z 1810401 [ExternalViewComputeWorker-TestClusterStatusMonitorLifecycle0_2] ERROR org.apache.helix.controller.pipeline.AbstractAsyncBaseStage - Failed to process DEFAULT::ExternalViewComputeStage asynchronously
2021-05-27T13:44:01.7127825Z java.lang.NullPointerException
2021-05-27T13:44:01.7130637Z at org.apache.helix.manager.zk.ZkBaseDataAccessor.set(ZkBaseDataAccessor.java:912)
2021-05-27T13:44:01.7135009Z at org.apache.helix.manager.zk.ZkBaseDataAccessor.setChildren(ZkBaseDataAccessor.java:856)
2021-05-27T13:44:01.7140459Z at org.apache.helix.manager.zk.ZKHelixDataAccessor.setChildren(ZKHelixDataAccessor.java:578)
2021-05-27T13:44:01.7146249Z at org.apache.helix.controller.stages.ExternalViewComputeStage.execute(ExternalViewComputeStage.java:134)
2021-05-27T13:44:01.7149517Z at org.apache.helix.controller.pipeline.AbstractAsyncBaseStage$1.run(AbstractAsyncBaseStage.java:47)
2021-05-27T13:44:01.7152294Z at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:701)
2021-05-27T13:44:01.7154919Z at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:697)
2021-05-27T13:44:01.7157110Z at org.apache.helix.common.DedupEventProcessor.run(DedupEventProcessor.java:61)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] github-actions[bot] commented on issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #1763:
URL: https://github.com/apache/helix/issues/1763#issuecomment-859197826
This test fails in: https://github.com/apache/helix/actions/runs/926889399
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang removed a comment on issue #1763: [Failed CI Test] testClusterStatusMonitorLifecycle(org.apache.helix.monitoring.TestClusterStatusMonitorLifecycle)
Posted by GitBox <gi...@apache.org>.
jiajunwang removed a comment on issue #1763:
URL: https://github.com/apache/helix/issues/1763#issuecomment-850690134
It seems that the following NPE blocks the metrics update (for cleaning up) so the metrics are not removed as expected.
```
2021-05-27T13:44:01.7121680Z 1810401 [ExternalViewComputeWorker-TestClusterStatusMonitorLifecycle0_2] ERROR org.apache.helix.controller.pipeline.AbstractAsyncBaseStage - Failed to process DEFAULT::ExternalViewComputeStage asynchronously
2021-05-27T13:44:01.7127825Z java.lang.NullPointerException
2021-05-27T13:44:01.7130637Z at org.apache.helix.manager.zk.ZkBaseDataAccessor.set(ZkBaseDataAccessor.java:912)
2021-05-27T13:44:01.7135009Z at org.apache.helix.manager.zk.ZkBaseDataAccessor.setChildren(ZkBaseDataAccessor.java:856)
2021-05-27T13:44:01.7140459Z at org.apache.helix.manager.zk.ZKHelixDataAccessor.setChildren(ZKHelixDataAccessor.java:578)
2021-05-27T13:44:01.7146249Z at org.apache.helix.controller.stages.ExternalViewComputeStage.execute(ExternalViewComputeStage.java:134)
2021-05-27T13:44:01.7149517Z at org.apache.helix.controller.pipeline.AbstractAsyncBaseStage$1.run(AbstractAsyncBaseStage.java:47)
2021-05-27T13:44:01.7152294Z at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:701)
2021-05-27T13:44:01.7154919Z at org.apache.helix.controller.GenericHelixController$3.handleEvent(GenericHelixController.java:697)
2021-05-27T13:44:01.7157110Z at org.apache.helix.common.DedupEventProcessor.run(DedupEventProcessor.java:61)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org