[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10231: (WIP) StarTree non deterministic ordering of functions in startree_index_map and other places

["Jackie-Jiang (via GitHub)" <gi...@apache.org>]

Jackie-Jiang commented on code in PR #10231:
URL: https://github.com/apache/pinot/pull/10231#discussion_r1098055170


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/startree/v2/builder/BaseSingleTreeBuilder.java:
##########
@@ -131,7 +132,7 @@ static class Record {
           "Dimension: " + dimension + " does not have dictionary");
     }
 
-    Set<AggregationFunctionColumnPair> functionColumnPairs = builderConfig.getFunctionColumnPairs();
+    Set<AggregationFunctionColumnPair> functionColumnPairs = new TreeSet<>(builderConfig.getFunctionColumnPairs());

Review Comment:
   This extra map is redundant as the builder config already create the pairs as `TreeSet`



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/startree/v2/store/StarTreeIndexMapUtils.java:
##########
@@ -112,6 +112,14 @@ public boolean equals(@Nullable Object obj) {
         return false;
       }
     }
+
+    @Override
+    public int compareTo(IndexKey other) {
+      return Comparator
+          .comparing((IndexKey i) -> i._column, Comparator.nullsLast(Comparator.naturalOrder()))
+          .thenComparing((IndexKey i) -> i._indexType, Comparator.nullsLast(Comparator.naturalOrder()))

Review Comment:
   (minor) index type cannot be null



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/startree/AggregationFunctionColumnPair.java:
##########
@@ -86,4 +87,12 @@ public boolean equals(Object obj) {
   public String toString() {
     return toColumnName();
   }
+
+  @Override
+  public int compareTo(AggregationFunctionColumnPair other) {
+    return Comparator.comparing((AggregationFunctionColumnPair o) -> o._column,
+            Comparator.nullsLast(Comparator.naturalOrder()))

Review Comment:
   (minor) neither of these 2 fields can be null



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/startree/v2/builder/StarTreeIndexCombiner.java:
##########
@@ -54,7 +54,7 @@ public StarTreeIndexCombiner(File indexFile)
    */
   public Map<IndexKey, IndexValue> combine(StarTreeV2BuilderConfig builderConfig, File starTreeIndexDir)
       throws IOException {
-    Map<IndexKey, IndexValue> indexMap = new HashMap<>();
+    Map<IndexKey, IndexValue> indexMap = new TreeMap<>();

Review Comment:
   Suggest making it a `List<Pair<IndexKey, IndexValue>>` instead of a `TreeMap` because we don't need to dedup



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org