[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #6869: Add segment size rule to Recommendation Engine

[GitBox <gi...@apache.org>]

mcvsubbu commented on a change in pull request #6869:
URL: https://github.com/apache/incubator-pinot/pull/6869#discussion_r625920258



##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/SegmentSizeRule.java
##########
@@ -85,7 +85,12 @@ public void run()
   }
 
   /**
-   * Estimate optimal segment size parameters
+   * Estimate segment size parameters by extrapolation based on the number of records and size of the generated segment.
+   * The linear extrapolation used here is not optimal because of columnar way of storing data and usage of different
+   * indices. Another way would be to iteratively generate new segments with expected number of rows until the ideal
+   * segment is found, but that's costly because of the time it takes to generate segments. Although the extrapolation
+   * approach seems to be less accurate, it is chosen due to its performance.

Review comment:
       How long does it take? Can we do 2 or 3 iterations, maybe?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org