Re: [I] Tool for migrating from one deep store to another [pinot]

["bgl-prahlad (via GitHub)" <gi...@apache.org>]

bgl-prahlad commented on issue #7275:
URL: https://github.com/apache/pinot/issues/7275#issuecomment-2451789832

   hi @npawar I would like to use this issue as an opportunity to get started with simple contributions. I would like some guidance in getting started. I can see that this issue is only solving a small part of the larger problem of migrating data-stores, and in that spirit I would like to address the usecase of migrating from a local store to a new location supported by PinotFS.
   
   I would like to create a new admin-command "SyncSegmentDeepStoreLocations" which would identify all segments in the cluster/controller whose location does not match the "controller.data.dir" configuration of the controller, and migrate these segments to the correct location - and update the ZK metadata.
   
   My initial thought for the workflow for migration is:
   - Pre-condition: Pinot Controller is operating with local store based configuration.
   - Steps: 
     - Update the controller configuration to update the deep-store location (controller.data.dir - eg see [Use S3 as Deep Storage for Pinot | Apache Pinot Docs](https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot))
     -  Run the “SyncSegmentDeepStoreLocations” admin command
   - Post-condition: If the SyncSegmentDeepStoreLocations succeeds, then all segments are migrated from their older location to the newly configured deep-store location (controller.data.dir)
   
   The "SyncSegmentDeepStoreLocations" command itself would:
   - Get the currently configured deep-store location
   - Iterate through all the tables and all segments (look at how ShowClusterInfoCommand does this)
   - Identify segments whose location does not match
   - Figure out the targetURI for the segment (follow logic for the CreateSegment command)
   - Instantiate PinotFS instance to copy file to the target location.
   
   Does this make sense?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org