You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/08/10 17:29:24 UTC
[GitHub] [pinot] sajjad-moradi opened a new issue #7274: Improve performance of realtime consumption status checker
sajjad-moradi opened a new issue #7274:
URL: https://github.com/apache/pinot/issues/7274
Realtime consumption status checker is used at startup time in Pinot Server to define a catch-up period in which no query execution happens and stream consumers try to catch up to the latest messages available in different streams. The current realtime consumption status checker defines a time threshold for startup consumption. When the wait time after Pinot Server starts up passes that time threshold, the status checker returns ServiceStatus.GOOD.
There are some ways to improve the existing behavior:
- Add consumption rate statistics to the StatsHistory files of consuming segments. Then in startup period, consume greedily until the consumption rate converges to the rate available in the StatsHistory file.
- At startup time, get the latest stream offset for each consuming segments and consume until the latest ingested offset is greater than the latest observed stream offset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] sajjad-moradi commented on issue #7274: Improve performance of realtime consumption status checker
Posted by GitBox <gi...@apache.org>.
sajjad-moradi commented on issue #7274:
URL: https://github.com/apache/pinot/issues/7274#issuecomment-896178167
Please note that this improvement has been a TODO for a long time:
https://github.com/apache/pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/utils/ServiceStatus.java#L204
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] sajjad-moradi commented on issue #7274: Improve performance of realtime consumption status checker
Posted by GitBox <gi...@apache.org>.
sajjad-moradi commented on issue #7274:
URL: https://github.com/apache/pinot/issues/7274#issuecomment-984218621
Implementation of the 2nd approach is done in #7267 and https://github.com/apache/pinot/pull/7753.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] sajjad-moradi closed issue #7274: Improve performance of realtime consumption status checker
Posted by GitBox <gi...@apache.org>.
sajjad-moradi closed issue #7274:
URL: https://github.com/apache/pinot/issues/7274
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] sajjad-moradi commented on issue #7274: Improve performance of realtime consumption status checker
Posted by GitBox <gi...@apache.org>.
sajjad-moradi commented on issue #7274:
URL: https://github.com/apache/pinot/issues/7274#issuecomment-896179063
The 2nd approach is being implemented in #7267
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org