[GitHub] [incubator-pinot] jackjlli commented on pull request #5922: Add max qps bucket count

[GitBox <gi...@apache.org>]

jackjlli commented on pull request #5922:
URL: https://github.com/apache/incubator-pinot/pull/5922#issuecomment-680647548


   > Shouldn't we be setting the value to "Max qps within a second since the time the callback was invoked last"? 
   
   We don't want to make the metrics system stateful. Every time the callback method gets called, it should return whatever the value should be.Thus, I don't think it's good to reset the counts when the callback gets called. Otherwise, we should have changed all the metrics to be stateful in Pinot cluster.
   
   > In some systems, the polling may be more often than 1 minute, and in others less often. So, we should keep a hit counter for some max time (say, 10m).
   
   I admit that the frequency of poll may vary. We can make it configurable. But the purpose of this new metric is to track the qps related statistics. If 10 mins is the granularity for a system to track qps, then I don't think they need to set qps quota for their tables.
   
   > The time interval could be close to a certain period, but can fluctuate by some small percentage either way. So, if the poll comes in 65 seconds, and our first second had a burst, we will lose it (as per your implementation).
   
   This is a rare case. In fact, what we are trying to solve is to find a way to detect the burst of queries which last for a while. They may be ignored, but will never be ignored all the time. Plus, if it happens quite often, then I think there is some issue on polling instead of adjusting our stateless metric system. The callback function never knows when it will be called in advance; when the callback function gets called, it should return the exact max qps within a minute based on the requirement. 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org