[GitHub] [incubator-pinot] npawar commented on issue #4218: Add RealtimeConsumptionCatchupServiceCallback

[GitBox <gi...@apache.org>]

npawar commented on issue #4218: Add RealtimeConsumptionCatchupServiceCallback
URL: https://github.com/apache/incubator-pinot/pull/4218#issuecomment-494139930
 
 
   > @npawar I have a few questions.
   > 
   > What is the impact on the latency and when does it kick in and for what kind of use cases?
   > How will the user know whats the right wait-time? We might still see the same issue if the server has not caught up within the wait time right?
   > 
   > One of the concerns here is that we stop the entire server from serving any other segment.
   > 
   > Looks like we need two things
   > 
   > * limit the number of threads for consumption so that it does not hog all the cpu during bootstrap/server startup
   > * Have the logic in the state transition for OFFLINE -> CONSUMING to not ack Helix until it's caught up. This has the added benefit that query results are up to date across all replicas.
   > 
   > I am ok with the temporary fix but was wondering what would be the right fix that requires minimal configuration from the user.
   
   The impact we've observed is usually for high ingestion rate realtime tables. The tables can usually take 2-3 minutes to completely catchup after server restart. During this time, we not only see high latencies/slow queries/timeouts because of cpu bottleneck, but also the results returned by that server will not be in sync with what other replicas return. 
   
   We decided to have a simple approach of static wait time to begin with, until we can get more information about the consumption rate. Yes we will still see the issue if the server has not caught up in the configured time. 
   We plan to include consumption rate related stats in server stats. Once we have those, we will remove the static wait. We will then wait until consumption rate stabilizes to the average rate or until idle.
   
   I can make the static wait to default 0 if having the default 5 is a problem?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org