[GitHub] [pinot] Jackie-Jiang commented on pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir

[GitBox <gi...@apache.org>]

Jackie-Jiang commented on PR #9306:
URL: https://github.com/apache/pinot/pull/9306#issuecomment-1239905005

   > When addOrReplaceSegment() is called, as there is no segment loaded yet, the segmentZKMetadata.tier is used to load the segment for the first time on server. When reloadSegment() is called, as the segment is loaded by server (and we're reloading it), the current tier tracked by the ImmutableSegment object is compared with segmentZKMetadata.tier to decide if there is need to move indexDir to a new dir before reloading the segment.
   
   What will happen if the tier is updated when server is restarting? Server won't be able to find the old local segment, and the old segment will left as orphan and never be cleaned up.
   I think we need to find a way to persist the current tier info to the disk so that when server restarts, it can find the local segment tier. Or we may search the previous tiers if we cannot find the local segment, and migrate the tier immediately if the local segment is in the data dir for another tier.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org