[GitHub] [pinot] Jackie-Jiang commented on pull request #6778: upload missing LLC segment to segment store by controller periodic task

[GitBox <gi...@apache.org>]

Jackie-Jiang commented on pull request #6778:
URL: https://github.com/apache/pinot/pull/6778#issuecomment-917293890


   @mcvsubbu The problem here is that there is already a known race condition needs to be fixed, and a lot of extra logic and extra config in order to avoid this zk metadata read. There are also bugs within the extra logic (e.g. uploaded segment whose name does not follow the LLC segment name format is not properly handled). IMO the extra effort of configuring the cluster and maintaining the code is not worth the gain.
   I would suggest just using the zk metadata and keeping the logic simple for the first version. We can consider reducing the frequency of the check if it becomes a bottleneck (e.g. follows the RealtimeSegmentValidationManager._segmentLevelValidationIntervalInSeconds).
   
   The periodic tasks that read all segment zk metadata:
   - RetentionManager (6 hours interval by default): both offline and realtime tables
   - RealtimeSegmentValidationManager (segment level validation has 24 hours interval by default): realtime tables
   - OfflineSegmentIntervalChecker (24 hours interval by default): offline tables


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org