[GitHub] [pinot] npawar commented on a change in pull request #7294: cleanup segment preprocessor a bit to make next RP smaller

[GitBox <gi...@apache.org>]

npawar commented on a change in pull request #7294:
URL: https://github.com/apache/pinot/pull/7294#discussion_r688193207



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentLoader.java
##########
@@ -169,7 +158,20 @@ public static ImmutableSegment load(File indexDir, IndexLoadingConfig indexLoadi
 
     ImmutableSegmentImpl segment =
         new ImmutableSegmentImpl(actualSegmentDirectory, segmentMetadata, indexContainerMap, starTreeIndexContainer);
-    LOGGER.info("Successfully loaded segment {} with config: {}", segmentName, segmentDirectoryLoaderConfigs);
+    LOGGER.info("Successfully loaded segment {} with config: {}", segmentName, segDirConfigs);
     return segment;
   }
+
+  // Pre-process the segment on local using local SegmentDirectory.
+  // Please note that this step may modify the segment metadata.
+  private static void preprocessSegment(File indexDir, IndexLoadingConfig indexLoadingConfig, Schema schema)
+      throws Exception {
+    PinotConfiguration tierConfigs = indexLoadingConfig.getTierConfigs();
+    PinotConfiguration segDirConfigs = new PinotConfiguration(tierConfigs.toMap());
+    SegmentDirectory segDir =
+        SegmentDirectoryLoaderRegistry.getLocalSegmentDirectoryLoader().load(indexDir.toURI(), segDirConfigs);

Review comment:
       you're right in terms of what the LocalSegmentDirectory needs. I choose to make the `getTierConfigs` be the call that returns all tier related properties. `loadMode` is a tier config for `local` tier. This way I could keep the `getLocalSegmentDirectory` call's arguments consistent across the tiers.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org