You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/05/17 20:50:05 UTC

[GitHub] [pinot] Jackie-Jiang opened a new pull request, #8721: Fix dimension table load when server restart or reload table

Jackie-Jiang opened a new pull request, #8721:
URL: https://github.com/apache/pinot/pull/8721

   Currently `DimensionTableDataManager` overrides the `addSegment(File indexDir, IndexLoadingConfig indexLoadingConfig)` method to reload the lookup table, but this method is not always invoked when adding the segment. Fix the issue by overriding `addSegment(ImmutableSegment immutableSegment)` which is always invoked when adding the segment.
   
   Other changes:
   - Enhance the validation for schema and primary key and fail with proper exception message
   - Enhance the test to mimic the actual flow of reloading the segment


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang merged pull request #8721: Fix dimension table load when server restart or reload table

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged PR #8721:
URL: https://github.com/apache/pinot/pull/8721


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] navina commented on a diff in pull request #8721: Fix dimension table load when server restart or reload table

Posted by GitBox <gi...@apache.org>.
navina commented on code in PR #8721:
URL: https://github.com/apache/pinot/pull/8721#discussion_r875355439


##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java:
##########
@@ -128,25 +128,26 @@ private void loadLookupTable()
     } while (!UPDATER.compareAndSet(this, snapshot, replacement));
   }
 
-  private DimensionTable createDimensionTable()
-      throws Exception {
-    Map<PrimaryKey, GenericRow> map = new HashMap<>();
+  private DimensionTable createDimensionTable() {
+    Schema schema = ZKMetadataProvider.getTableSchema(_propertyStore, _tableNameWithType);
+    Preconditions.checkState(schema != null, "Failed to find schema for dimension table: %s", _tableNameWithType);
+    List<String> primaryKeyColumns = schema.getPrimaryKeyColumns();
+    Preconditions.checkState(CollectionUtils.isNotEmpty(primaryKeyColumns),
+        "Primary key columns must be configured for dimension table: %s", _tableNameWithType);
+
+    Map<PrimaryKey, GenericRow> lookupTable = new HashMap<>();
     List<SegmentDataManager> segmentManagers = acquireAllSegments();
     try {
       for (SegmentDataManager segmentManager : segmentManagers) {
         IndexSegment indexSegment = segmentManager.getSegment();
-        try (PinotSegmentRecordReader reader = new PinotSegmentRecordReader(
-            indexSegment.getSegmentMetadata().getIndexDir())) {
-          while (reader.hasNext()) {
-            GenericRow row = reader.next();
-            map.put(row.getPrimaryKey(_dimensionTable.getPrimaryKeyColumns()), row);
-          }
+        int numTotalDocs = indexSegment.getSegmentMetadata().getTotalDocs();

Review Comment:
   Makes sense. Thanks! 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8721: Fix dimension table load when server restart or reload table

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #8721:
URL: https://github.com/apache/pinot/pull/8721#discussion_r875303678


##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java:
##########
@@ -128,25 +128,26 @@ private void loadLookupTable()
     } while (!UPDATER.compareAndSet(this, snapshot, replacement));
   }
 
-  private DimensionTable createDimensionTable()
-      throws Exception {
-    Map<PrimaryKey, GenericRow> map = new HashMap<>();
+  private DimensionTable createDimensionTable() {
+    Schema schema = ZKMetadataProvider.getTableSchema(_propertyStore, _tableNameWithType);
+    Preconditions.checkState(schema != null, "Failed to find schema for dimension table: %s", _tableNameWithType);
+    List<String> primaryKeyColumns = schema.getPrimaryKeyColumns();
+    Preconditions.checkState(CollectionUtils.isNotEmpty(primaryKeyColumns),
+        "Primary key columns must be configured for dimension table: %s", _tableNameWithType);
+
+    Map<PrimaryKey, GenericRow> lookupTable = new HashMap<>();
     List<SegmentDataManager> segmentManagers = acquireAllSegments();
     try {
       for (SegmentDataManager segmentManager : segmentManagers) {
         IndexSegment indexSegment = segmentManager.getSegment();
-        try (PinotSegmentRecordReader reader = new PinotSegmentRecordReader(
-            indexSegment.getSegmentMetadata().getIndexDir())) {
-          while (reader.hasNext()) {
-            GenericRow row = reader.next();
-            map.put(row.getPrimaryKey(_dimensionTable.getPrimaryKeyColumns()), row);
-          }
+        int numTotalDocs = indexSegment.getSegmentMetadata().getTotalDocs();

Review Comment:
   Good question. We don't want to create a separate record reader because that will load the segment again. Instead, we can directly read the record out of the already loaded segment



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #8721: Fix dimension table load when server restart or reload table

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #8721:
URL: https://github.com/apache/pinot/pull/8721#issuecomment-1129348268

   # [Codecov](https://codecov.io/gh/apache/pinot/pull/8721?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#8721](https://codecov.io/gh/apache/pinot/pull/8721?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5c84901) into [master](https://codecov.io/gh/apache/pinot/commit/3bbae6ef6145ca29af7f8d0f115263437eef63bd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3bbae6e) will **increase** coverage by `3.41%`.
   > The diff coverage is `76.00%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #8721      +/-   ##
   ============================================
   + Coverage     66.20%   69.62%   +3.41%     
   - Complexity     4389     4576     +187     
   ============================================
     Files          1279     1729     +450     
     Lines         64619    90183   +25564     
     Branches      10089    13415    +3326     
   ============================================
   + Hits          42781    62786   +20005     
   - Misses        18802    23044    +4242     
   - Partials       3036     4353    +1317     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `26.96% <0.00%> (?)` | |
   | integration2 | `25.34% <0.00%> (?)` | |
   | unittests1 | `66.13% <76.00%> (-0.07%)` | :arrow_down: |
   | unittests2 | `14.27% <0.00%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8721?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ata/manager/offline/DimensionTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvb2ZmbGluZS9EaW1lbnNpb25UYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `82.14% <76.00%> (-2.48%)` | :arrow_down: |
   | [...ore/query/scheduler/resources/ResourceManager.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9zY2hlZHVsZXIvcmVzb3VyY2VzL1Jlc291cmNlTWFuYWdlci5qYXZh) | `84.00% <0.00%> (-12.00%)` | :arrow_down: |
   | [.../impl/dictionary/BaseOffHeapMutableDictionary.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvQmFzZU9mZkhlYXBNdXRhYmxlRGljdGlvbmFyeS5qYXZh) | `81.33% <0.00%> (-3.34%)` | :arrow_down: |
   | [.../pinot/core/query/scheduler/PriorityScheduler.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9zY2hlZHVsZXIvUHJpb3JpdHlTY2hlZHVsZXIuamF2YQ==) | `78.08% <0.00%> (-2.74%)` | :arrow_down: |
   | [...e/pinot/segment/local/io/util/PinotDataBitSet.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pby91dGlsL1Bpbm90RGF0YUJpdFNldC5qYXZh) | `95.62% <0.00%> (-1.46%)` | :arrow_down: |
   | [...org/apache/pinot/core/common/ObjectSerDeUtils.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vT2JqZWN0U2VyRGVVdGlscy5qYXZh) | `89.57% <0.00%> (-0.82%)` | :arrow_down: |
   | [.../aggregation/function/ModeAggregationFunction.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9Nb2RlQWdncmVnYXRpb25GdW5jdGlvbi5qYXZh) | `87.56% <0.00%> (-0.82%)` | :arrow_down: |
   | [...he/pinot/segment/local/utils/SegmentPushUtils.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9TZWdtZW50UHVzaFV0aWxzLmphdmE=) | `12.70% <0.00%> (-0.22%)` | :arrow_down: |
   | [...ingestion/batch/spec/SegmentGenerationJobSpec.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvaW5nZXN0aW9uL2JhdGNoL3NwZWMvU2VnbWVudEdlbmVyYXRpb25Kb2JTcGVjLmphdmE=) | `53.33% <0.00%> (ø)` | |
   | [...ent/local/data/manager/TableDataManagerConfig.java](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9kYXRhL21hbmFnZXIvVGFibGVEYXRhTWFuYWdlckNvbmZpZy5qYXZh) | `0.00% <0.00%> (ø)` | |
   | ... and [688 more](https://codecov.io/gh/apache/pinot/pull/8721/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8721?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8721?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [3bbae6e...5c84901](https://codecov.io/gh/apache/pinot/pull/8721?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] navina commented on a diff in pull request #8721: Fix dimension table load when server restart or reload table

Posted by GitBox <gi...@apache.org>.
navina commented on code in PR #8721:
URL: https://github.com/apache/pinot/pull/8721#discussion_r875264972


##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java:
##########
@@ -128,25 +128,26 @@ private void loadLookupTable()
     } while (!UPDATER.compareAndSet(this, snapshot, replacement));
   }
 
-  private DimensionTable createDimensionTable()
-      throws Exception {
-    Map<PrimaryKey, GenericRow> map = new HashMap<>();
+  private DimensionTable createDimensionTable() {
+    Schema schema = ZKMetadataProvider.getTableSchema(_propertyStore, _tableNameWithType);
+    Preconditions.checkState(schema != null, "Failed to find schema for dimension table: %s", _tableNameWithType);
+    List<String> primaryKeyColumns = schema.getPrimaryKeyColumns();
+    Preconditions.checkState(CollectionUtils.isNotEmpty(primaryKeyColumns),
+        "Primary key columns must be configured for dimension table: %s", _tableNameWithType);
+
+    Map<PrimaryKey, GenericRow> lookupTable = new HashMap<>();
     List<SegmentDataManager> segmentManagers = acquireAllSegments();
     try {
       for (SegmentDataManager segmentManager : segmentManagers) {
         IndexSegment indexSegment = segmentManager.getSegment();
-        try (PinotSegmentRecordReader reader = new PinotSegmentRecordReader(
-            indexSegment.getSegmentMetadata().getIndexDir())) {
-          while (reader.hasNext()) {
-            GenericRow row = reader.next();
-            map.put(row.getPrimaryKey(_dimensionTable.getPrimaryKeyColumns()), row);
-          }
+        int numTotalDocs = indexSegment.getSegmentMetadata().getTotalDocs();

Review Comment:
   qq: Why not continue using the `PinotSegmentRecordReader` ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org