[GitHub] [pinot] jonminter commented on issue #8618: Query Processing Resiliency

[GitBox <gi...@apache.org>]

jonminter commented on issue #8618:
URL: https://github.com/apache/pinot/issues/8618#issuecomment-1235845525

   So my coworker just pointed me to this issue. I'm an engineer at Stripe and we've just started experimenting with some similar ideas for making Pinot reliable. One thing I had just implemented in our fork and was testing today was a rate limiter for the Pinot broker that measures observed query latency and estimates server queue sizes using the TCP Vegas congestion control algorithm. I'm using this [rate limiter implementation from Netflix](https://github.com/Netflix/concurrency-limits).
   
   I've only just started testing this but I can provide some data once I've had a chance to do some more comprehensive tests. But initial tests it looks promising.
   
   I had similar thoughts about measuring server latencies and implementing load balanced routing and looks like you guys have already started implementing that which is great to see! We have observed similar issues in our clusters with a server that has degraded performance affecting overall latency.
   
   Also had some ideas about how to meter resource usage on the servers during rebalances as we've noticed this can impact query performance as well.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org