[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #5875: Config recommender patch

[GitBox <gi...@apache.org>]

jackjlli commented on a change in pull request #5875:
URL: https://github.com/apache/incubator-pinot/pull/5875#discussion_r473276920



##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/utils/QueryInvertedSortedIndexRecommender.java
##########
@@ -585,7 +573,9 @@ else if (type == Predicate.Type.IN || type == Predicate.Type.NOT_IN) {
 
       boolean isFirst = false;
       List<String> values = (type == Predicate.Type.IN)?((InPredicate) predicate).getValues():((NotInPredicate) predicate).getValues();
-      if (values.get(RecommenderConstants.FIRST).equals(RecommenderConstants.IN_PREDICATE_ESTIMATE_LEN_FLAG) ||
+      if(values.size()==1){

Review comment:
       should put a space between `)` and `{`

##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/io/params/RecommenderConstants.java
##########
@@ -32,7 +32,7 @@
     public static final double DEFAULT_THRESHOLD_MIN_AND_PREDICATE_INCREMENTAL_VOTE = 0.6d;
     public static final double DEFAULT_THRESHOLD_RATIO_MIN_AND_PREDICATE_TOP_CANDIDATES = 0.8d;
     public static final double DEFAULT_THRESHOLD_RATIO_MIN_GAIN_DIFF_BETWEEN_ITERATION = 0.05d;
-    public static final int DEFAULT_MAX_NUM_ITERATION_WITHOUT_GAIN = 3;
+    public static final int DEFAULT_MAX_NUM_ITERATION_WITHOUT_GAIN = 2;

Review comment:
       Can we add the link of this class to the doc so that we know this is the default values?

##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/InvertedSortedIndexJointRule.java
##########
@@ -197,28 +194,48 @@ public PredicateParseResult findOptimalCombination(List<List<PredicateParseResul
    * {@link PredicateParseResult#getnESI()} the nESI before applying any index
    * {@link PredicateParseResult#getnESIWithIdx()} the estimated nESI after applying optimal indices
    */
-  public PredicateParseResult evaluateCombination(int n, int r, List<List<PredicateParseResult>> parsedQuery) {
-    List<int[]> combinationIntArrays = generateCombinations(n, r);
-    LOGGER.debug("combinationIntArrays {}", combinationIntArrays);
+  public PredicateParseResult evaluateCombination(final int n, final int r, List<List<PredicateParseResult>> parsedQuery) {
+    FixedLenBitset usedCols = new FixedLenBitset(n);
+    parsedQuery.forEach(list -> list.stream()
+        .filter(predicateParseResult -> (predicateParseResult.getCandidateDims().getCardinality() <= r))
+        .forEach(predicateParseResult -> usedCols.union(predicateParseResult.getCandidateDims())));
+    LOGGER.debug("totalUsed {}", usedCols.getCardinality());
+
+    List<Integer> usedColIDs = usedCols.getOffsets();
+    int nCapped = usedColIDs.size();
+    int rCapped = Math.min(r, nCapped);
+
+    int[] idToColID = new int[nCapped];
+    for (int i = 0; i < nCapped; i++) {
+      idToColID[i] = usedColIDs.get(i);
+    }
+
 
     // Enumerate all possible combinations of r-sized set, which will be applied indices on
-    List<FixedLenBitset> combinations = combinationIntArrays.parallelStream().map(combinationIntArray -> {
-      FixedLenBitset indices = new FixedLenBitset(n);
-      for (int j = 0; j < r; j++) {
-        indices.add(combinationIntArray[j]);
-      }
-      return indices;
-    }).collect(Collectors.toList());
+    List<int[]> combinationIntArrays = generateCombinations(nCapped, rCapped);
+
+//     Enumerate all possible combinations of r-sized set, which will be applied indices on
+//    List<FixedLenBitset> combinations = combinationIntArrays.parallelStream().map(combinationIntArray -> {

Review comment:
       Maybe we should remove this comments?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org