[GitHub] [incubator-pinot] mcvsubbu commented on issue #4345: Config to make realtime non-winner servers download the segment instead of build

[GitBox <gi...@apache.org>]

mcvsubbu commented on issue #4345: Config to make realtime non-winner servers download the segment instead of build
URL: https://github.com/apache/incubator-pinot/issues/4345#issuecomment-504108545
 
 
   @kishoreg making the wait-time configurable will not help if the consuming thread has already started building the segment. It cannot be interrupted when the transition is received,and the only option is to wait it out.
   Jackie's change in #4338 is being evaluated in production, and yes, if it works, this change is not needed for performance. But it *is* needed for freshness. Otherwise, for a use case that takes order of  mins to build a segment we will be serving stale data in the meantime.
   This change has already been tried out in production, and the results are promising.
   
   Yes, it will create additional load on the controllers (actually, on the storage). That is why this is a table configuration, to be turned on in tables on a need-basis that satisfy multiple conditions. In our produciton scenarios. we have not felt the need for any table other than one, for this feature. The one table has 10 replicas, a very high qps and low latency requirement (< 100ms in 99th percentile), low memory foot-print, and takes minutes to build segments. We have tried out other things like building segments sooner, etc. didn't work.
   
   The idea here is to let the controller decide whether a server should be downoading the segment or building one. In the initial version it will be a brute-force "DISCARD" return from the controller, which causes the server to just download the segment. In the medium term (or even in the first PR) we can have the controller return KEEP for 50% of the hosts, and DISCARD for the other 50%. In future, we can make more intelligent decisions.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org