[GitHub] [pinot] KKcorps commented on a diff in pull request #9802: Add memory optimized dimension table

[GitBox <gi...@apache.org>]

KKcorps commented on code in PR #9802:
URL: https://github.com/apache/pinot/pull/9802#discussion_r1032761596


##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java:
##########
@@ -156,7 +178,43 @@ private DimensionTable createDimensionTable() {
           }
         }
       }
-      return new DimensionTable(schema, primaryKeyColumns, lookupTable);
+        return new FastLookupDimensionTable(schema, primaryKeyColumns, lookupTable);
+    } finally {
+      for (SegmentDataManager segmentManager : segmentManagers) {
+        releaseSegment(segmentManager);
+      }
+    }
+  }
+
+  private DimensionTable createMemOptimisedDimensionTable() {
+    Schema schema = ZKMetadataProvider.getTableSchema(_propertyStore, _tableNameWithType);
+    Preconditions.checkState(schema != null, "Failed to find schema for dimension table: %s", _tableNameWithType);
+
+    List<String> primaryKeyColumns = schema.getPrimaryKeyColumns();
+    Preconditions.checkState(CollectionUtils.isNotEmpty(primaryKeyColumns),
+        "Primary key columns must be configured for dimension table: %s", _tableNameWithType);
+
+    Map<PrimaryKey, LookupRecordLocation> lookupTable = new HashMap<>();
+    List<SegmentDataManager> segmentManagers = acquireAllSegments();
+    try {
+      for (SegmentDataManager segmentManager : segmentManagers) {
+        IndexSegment indexSegment = segmentManager.getSegment();
+        int numTotalDocs = indexSegment.getSegmentMetadata().getTotalDocs();
+        if (numTotalDocs > 0) {
+          try (PinotSegmentRecordReader recordReader = new PinotSegmentRecordReader()) {

Review Comment:
   My bad. I have changed the logic now to close the segments only when shutting down the table data manager. The record reader is closed when we replace the table we newer instance.  Both of these happen only when preload is disabled, otherwise old logic is applied.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org