[GitHub] [pinot] klsince commented on pull request #7898: Add help methods to check if segment needs reprocessing

[GitBox <gi...@apache.org>]

klsince commented on pull request #7898:
URL: https://github.com/apache/pinot/pull/7898#issuecomment-993213880


   > I don't fully understand the optimization here. Looks like the new flow will be `ImmutableSegmentLoader` will call `SegmentPReprocessor.needsPreprocess()` and ...
   
   Thanks for the comments. Many of your observations are right. This refactoring is mainly driven by the need to make the logic related to segment loading also work with remote tier backend efficiently (a SegmentDirectory implementation different from SegmentLocalFSDirectory); and meanwhile make the logic easy to extend by assuming interface instead of a concrete local index dir.
   
   In fact, it's BaseTableDataManager's addOrReplaceSegment() and reloadSegment() methods to be refactored to call the new needProcess() method. They'll handle cases where SegmentDirectory can be null. Both methods will also be refactored to work with interfaces (i.e. SegmentDirectory and SegmentDirectoryLoader) instead of directly assuming a local index dir.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org