You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/07/28 21:29:54 UTC
[GitHub] [pinot] MeihanLi opened a new issue, #9128: [Alerting/Monitoring] Add an API to get the server shutDownInProgress config and expose the info at tenant level.
MeihanLi opened a new issue, #9128:
URL: https://github.com/apache/pinot/issues/9128
Can we add an API to get the server shutDownInProgress config and expose the info at tenant level?
We recently saw that some upsert servers can become unresponsive and stop serving any queries after a server restart. Even the server is healthy, the shutDownInProgress flag can not be set back to false (ready to serve queries). This caused an incident on our side and it took us some time to dig into the broker logs to found out that some servers became unresponsive for a long time.
It would be useful if we can add an API to get server shutDownInProgress config and expose the info at tenant level so that we can proper alerting to avoid such incidents again.
related pr: https://github.com/apache/pinot/pull/8525
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] MeihanLi commented on issue #9128: [Alerting/Monitoring] Add an API to get the server shutDownInProgress config and expose the info at tenant level.
Posted by GitBox <gi...@apache.org>.
MeihanLi commented on issue #9128:
URL: https://github.com/apache/pinot/issues/9128#issuecomment-1198653362
cc: @yupeng9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9128: [Alerting/Monitoring] Add an API to get the server shutDownInProgress config and expose the info at tenant level.
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9128:
URL: https://github.com/apache/pinot/issues/9128#issuecomment-1201708269
We can definitely add an API to return whether server is ready to serve queries (in addition to `shutDownInProgress` flag, we might also want to track `HELIX_ENABLED` which indicates if server is enabled). These info are stored in the `InstanceConfig` of the server in ZK.
Can you elaborate more on exposing the info at tenant level? You mean returning the percentage of servers that can serve queries?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org