You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2022/04/17 19:30:52 UTC

[GitHub] [helix] vaidhyanathan-ananthakrishnan-13 opened a new issue, #2048: ExternalViewChangeListener not getting called randomly for big cluster size.

vaidhyanathan-ananthakrishnan-13 opened a new issue, #2048:
URL: https://github.com/apache/helix/issues/2048

   ### Describe the bug
   We are having a few hundred listeners as spectators for the helix cluster and thousand+ of participants in the cluster. But when there is a change in the cluster, not all the listeners are getting called. sometimes, 5-10 are getting called, sometimes none, and the majority of the time all the spectators are getting called. Using ExternalViewChangeListener to register 
   
   The callback method onExternalViewChange
   
   ### To Reproduce
   Having a huge Cluster size affects the callback of the ExternalViewChangeListener
   
   ### Expected behavior
   All the spectators should get a callback for all the events. 
   
   ### Additional context
   It spectators are unreliable because of this bug.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.

Posted by GitBox <gi...@apache.org>.
desaikomal commented on issue #2048:
URL: https://github.com/apache/helix/issues/2048#issuecomment-1104329842

   not that i am aware of. i am relatively new and so not sure if others have seen this issue before.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] narendly commented on issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.

Posted by GitBox <gi...@apache.org>.
narendly commented on issue #2048:
URL: https://github.com/apache/helix/issues/2048#issuecomment-1104332789

   @vaidhyanathan-ananthakrishnan-13 Yes, this has been observed, but it's more that even though the cluster state has been updated, we're sometimes missing the corresponding updates to ExternalView (aka ExternalView doesn't get updated properly), which would lead to ExternalViewChangeListener not getting called. We have seen this in large-scale clusters.
   
   We are tracking this issue, but feel free to add any findings or suggestions here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] desaikomal commented on issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.

Posted by GitBox <gi...@apache.org>.
desaikomal commented on issue #2048:
URL: https://github.com/apache/helix/issues/2048#issuecomment-1101466013

   Hello @vaidhyanathan-ananthakrishnan-13 - can you elaborate on what is the size of the cluster, #resources etc this will help us simulate 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] vaidhyanathan-ananthakrishnan-13 commented on issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.

Posted by GitBox <gi...@apache.org>.
vaidhyanathan-ananthakrishnan-13 commented on issue #2048:
URL: https://github.com/apache/helix/issues/2048#issuecomment-1104227063

   @desaikomal 
   
   Does any known issue exist for this case? Or is it fine to add some kind of fallback mechanism for this(Not sure how to do it effectively without polling the state)? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] vaidhyanathan-ananthakrishnan-13 commented on issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.

Posted by GitBox <gi...@apache.org>.
vaidhyanathan-ananthakrishnan-13 commented on issue #2048:
URL: https://github.com/apache/helix/issues/2048#issuecomment-1104631926

   Thanks, @narendly  , 
   
   I will try to find out why the ExternalView is not getting updated.
   
   Meanwhile, as a  workaround, Is there any way we can know about the cluster state deterministically from the spectator? 
   
   One thought, I had is to add the `addLiveInstanceChangeListener` and get the cluster state from the spectator, but I am not sure really sure how to get the cluster state without ExternalView, Any help here? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] vaidhyanathan-ananthakrishnan-13 commented on issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.

Posted by GitBox <gi...@apache.org>.
vaidhyanathan-ananthakrishnan-13 commented on issue #2048:
URL: https://github.com/apache/helix/issues/2048#issuecomment-1101544619

   @desaikomal  Sorry wrongly closed the issue. 
   
   The size of the cluster is ~2500  instances and the listener is around ~1400. We are having two kinds of listener, one is 1400 another is ~100. 
   
   ExternalViewChangeListener is around ~1400 instances. but for that, This issue happening. 
   
   If I remember correctly, The same code base with the same cluster works fine, but over time, had to add a lot of instances, after some time started seeing this error.  I am not 100% accurate about whether it worked perfectly fine before or not, but now days see a lot more issues. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [I] ExternalViewChangeListener not getting called randomly for big cluster size. [helix]

Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue closed issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.
URL: https://github.com/apache/helix/issues/2048


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] vaidhyanathan-ananthakrishnan-13 closed issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.

Posted by GitBox <gi...@apache.org>.
vaidhyanathan-ananthakrishnan-13 closed issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.
URL: https://github.com/apache/helix/issues/2048


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [I] ExternalViewChangeListener not getting called randomly for big cluster size. [helix]

Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on issue #2048:
URL: https://github.com/apache/helix/issues/2048#issuecomment-2278692201

   ZK guarantee the sequence of listener callback for a same connection session. But no guarantee for cross session calls. As you described, you would like to get sequential calls from different spectator which is almost impossible.
   
   The reason is when your spectator connects to ZK, they may connect to different quorum node / observers. 
   
   Let me know if you would like to have anything for Helix side can help.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [I] ExternalViewChangeListener not getting called randomly for big cluster size. [helix]

Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on issue #2048:
URL: https://github.com/apache/helix/issues/2048#issuecomment-2284674350

   Feel free to reopen it if you have further questions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org