You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2022/04/17 19:30:52 UTC
[GitHub] [helix] vaidhyanathan-ananthakrishnan-13 opened a new issue, #2048: ExternalViewChangeListener not getting called randomly for big cluster size.
vaidhyanathan-ananthakrishnan-13 opened a new issue, #2048:
URL: https://github.com/apache/helix/issues/2048
### Describe the bug
We are having a few hundred listeners as spectators for the helix cluster and thousand+ of participants in the cluster. But when there is a change in the cluster, not all the listeners are getting called. sometimes, 5-10 are getting called, sometimes none, and the majority of the time all the spectators are getting called. Using ExternalViewChangeListener to register
The callback method onExternalViewChange
### To Reproduce
Having a huge Cluster size affects the callback of the ExternalViewChangeListener
### Expected behavior
All the spectators should get a callback for all the events.
### Additional context
It spectators are unreliable because of this bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] desaikomal commented on issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.
Posted by GitBox <gi...@apache.org>.
desaikomal commented on issue #2048:
URL: https://github.com/apache/helix/issues/2048#issuecomment-1104329842
not that i am aware of. i am relatively new and so not sure if others have seen this issue before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] narendly commented on issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.
Posted by GitBox <gi...@apache.org>.
narendly commented on issue #2048:
URL: https://github.com/apache/helix/issues/2048#issuecomment-1104332789
@vaidhyanathan-ananthakrishnan-13 Yes, this has been observed, but it's more that even though the cluster state has been updated, we're sometimes missing the corresponding updates to ExternalView (aka ExternalView doesn't get updated properly), which would lead to ExternalViewChangeListener not getting called. We have seen this in large-scale clusters.
We are tracking this issue, but feel free to add any findings or suggestions here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] desaikomal commented on issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.
Posted by GitBox <gi...@apache.org>.
desaikomal commented on issue #2048:
URL: https://github.com/apache/helix/issues/2048#issuecomment-1101466013
Hello @vaidhyanathan-ananthakrishnan-13 - can you elaborate on what is the size of the cluster, #resources etc this will help us simulate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] vaidhyanathan-ananthakrishnan-13 commented on issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.
Posted by GitBox <gi...@apache.org>.
vaidhyanathan-ananthakrishnan-13 commented on issue #2048:
URL: https://github.com/apache/helix/issues/2048#issuecomment-1104227063
@desaikomal
Does any known issue exist for this case? Or is it fine to add some kind of fallback mechanism for this(Not sure how to do it effectively without polling the state)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] vaidhyanathan-ananthakrishnan-13 commented on issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.
Posted by GitBox <gi...@apache.org>.
vaidhyanathan-ananthakrishnan-13 commented on issue #2048:
URL: https://github.com/apache/helix/issues/2048#issuecomment-1104631926
Thanks, @narendly ,
I will try to find out why the ExternalView is not getting updated.
Meanwhile, as a workaround, Is there any way we can know about the cluster state deterministically from the spectator?
One thought, I had is to add the `addLiveInstanceChangeListener` and get the cluster state from the spectator, but I am not sure really sure how to get the cluster state without ExternalView, Any help here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] vaidhyanathan-ananthakrishnan-13 commented on issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.
Posted by GitBox <gi...@apache.org>.
vaidhyanathan-ananthakrishnan-13 commented on issue #2048:
URL: https://github.com/apache/helix/issues/2048#issuecomment-1101544619
@desaikomal Sorry wrongly closed the issue.
The size of the cluster is ~2500 instances and the listener is around ~1400. We are having two kinds of listener, one is 1400 another is ~100.
ExternalViewChangeListener is around ~1400 instances. but for that, This issue happening.
If I remember correctly, The same code base with the same cluster works fine, but over time, had to add a lot of instances, after some time started seeing this error. I am not 100% accurate about whether it worked perfectly fine before or not, but now days see a lot more issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] ExternalViewChangeListener not getting called randomly for big cluster size. [helix]
Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue closed issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.
URL: https://github.com/apache/helix/issues/2048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] vaidhyanathan-ananthakrishnan-13 closed issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.
Posted by GitBox <gi...@apache.org>.
vaidhyanathan-ananthakrishnan-13 closed issue #2048: ExternalViewChangeListener not getting called randomly for big cluster size.
URL: https://github.com/apache/helix/issues/2048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] ExternalViewChangeListener not getting called randomly for big cluster size. [helix]
Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on issue #2048:
URL: https://github.com/apache/helix/issues/2048#issuecomment-2278692201
ZK guarantee the sequence of listener callback for a same connection session. But no guarantee for cross session calls. As you described, you would like to get sequential calls from different spectator which is almost impossible.
The reason is when your spectator connects to ZK, they may connect to different quorum node / observers.
Let me know if you would like to have anything for Helix side can help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] ExternalViewChangeListener not getting called randomly for big cluster size. [helix]
Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on issue #2048:
URL: https://github.com/apache/helix/issues/2048#issuecomment-2284674350
Feel free to reopen it if you have further questions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org