[GitHub] [pinot] yupeng9 commented on pull request #6291: Add consumption rate limiter for LLConsumer

[GitBox <gi...@apache.org>]

yupeng9 commented on pull request #6291:
URL: https://github.com/apache/pinot/pull/6291#issuecomment-1023411073


   > > +1 to this feature.
   > > Does the current implementation allow dynamic rate limit addition or change? For ops, it would be nice to adjust the quota and expect it to be effective immediately. Currently implementation seems to create the rate limiter once upon the `LLRealtimeSegmentDataManager ` creation
   > 
   > You're right, in this implementation changes in the ingestion quota config doesn't get reflected in the current consuming segments and we need to wait till they get completed and then the new consuming segments will pick up the new rate limit.
   > 
   > For immediate reflection of rate limit change, we can add ZK watches on table config or read the table configs in short periods of time like in one minute. Then adjust the rate limit if it's changed. But I think using these approaches adds complexity and maybe extra pressure on ZK. One simpler and less expensive approach is that after consumption rate limit is changed, the operator calls a force commit endpoint on controllers for existing consuming segments. Then the next consuming segments will automatically pick up the new consumption quota value. I'm finalizing a design doc for pause / resume consumption which includes the force commit logic and will share it soon.
   > 
   > @yupeng9 @mcvsubbu what do you guys think?
   
   Yes, I think this is a good solution. In fact, some other config changes such as schema evolution may follow a similar approach of force closing the consuming segments, as those changes also encounter similar challenges of waiting for the consuming segment to close.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org