[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10905: Startree index build enhancement

["Jackie-Jiang (via GitHub)" <gi...@apache.org>]

Jackie-Jiang commented on code in PR #10905:
URL: https://github.com/apache/pinot/pull/10905#discussion_r1232927284


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java:
##########
@@ -243,10 +243,8 @@ private void processStarTrees(File indexDir)
       if (starTreeMetadataList != null) {
         // There are existing star-trees
         if (StarTreeBuilderUtils.shouldRemoveExistingStarTrees(starTreeBuilderConfigs, starTreeMetadataList)) {

Review Comment:
   (minor) Let's rename this method to something like `shouldModifyExistingStarTrees()` and also modify the documentation to reflect the change



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java:
##########
@@ -243,10 +243,8 @@ private void processStarTrees(File indexDir)
       if (starTreeMetadataList != null) {
         // There are existing star-trees
         if (StarTreeBuilderUtils.shouldRemoveExistingStarTrees(starTreeBuilderConfigs, starTreeMetadataList)) {
-          // Remove the existing star-trees
-          LOGGER.info("Removing star-trees from segment: {}", _segmentMetadata.getName());
-          StarTreeBuilderUtils.removeStarTrees(indexDir);
-          _segmentMetadata = new SegmentMetadataImpl(indexDir);
+          LOGGER.info("Change detected in star-trees for segment: {}", _segmentMetadata.getName());
+          shouldGenerateStarTree = true;

Review Comment:
   We can simply remove the existing star trees when `shouldGenerateStarTree` is false (user wants to remove all the star-trees)



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/startree/v2/builder/MultipleTreesBuilder.java:
##########
@@ -115,13 +117,33 @@ public MultipleTreesBuilder(@Nullable List<StarTreeIndexConfig> indexConfigs, bo
     }
   }
 
+  private SeparatedStarTreesMetadata getExistingStarTrees()
+      throws Exception {
+    try {
+      if (_metadataProperties.containsKey(MetadataKey.STAR_TREE_COUNT)) {
+        // Extract existing startrees
+        // clean star-tree related files and configs once all the star-trees are separated and extracted
+        SeparatedStarTreesMetadata existingStarTrees = extractStarTrees();
+        StarTreeBuilderUtils.removeStarTrees(_segmentDirectory);

Review Comment:
   `removeStarTrees()` should take `indexDir` instead of `segmentDirectory` (even though it works if we pass in the `segmentDirectory`). `segmentDirectory` could be the `v3` sub-directory under the `indexDir`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org