You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "vvivekiyer (via GitHub)" <gi...@apache.org> on 2024/01/12 20:03:34 UTC

[PR] Bugfix to load dictionary onHeap when defined in fieldConfigList [pinot]

vvivekiyer opened a new pull request, #12265:
URL: https://github.com/apache/pinot/pull/12265

   label = bugfix
   
    
   PR to fix issue -> https://github.com/apache/pinot/issues/12254 
   
   Copying the description from the issue here. 
   
   ### Issue
   **Table Config**
   
   ```
   {
     "tableName": "customer_OFFLINE",
     "tableType": "OFFLINE",
     "segmentsConfig": {
       "replication": "1",
       "segmentPushType": "REFRESH",
       "schemaName": "customer",
       "minimizeDataMovement": false
     },
     "tenants": {},
     "tableIndexConfig": {
       "autoGeneratedInvertedIndex": false,
       "enableDynamicStarTreeCreation": false,
       "columnMajorSegmentBuilderEnabled": false,
       "optimizeDictionaryForMetrics": false,
       "noDictionarySizeRatioThreshold": 0.85,
       "rangeIndexVersion": 2,
       "sortedColumn": [
         "C_NAME"
       ],
       "loadMode": "HEAP",
       "enableDefaultStarTree": false,
       "aggregateMetrics": false,
       "nullHandlingEnabled": false,
       "optimizeDictionary": false,
       "createInvertedIndexDuringSegmentGeneration": false
     },
     "metadata": {
       "customConfigs": {}
     },
     "fieldConfigList": [
       {
         "name": "C_NAME",
         "encodingType": "DICTIONARY",
         "indexTypes": [],
         "indexes": {
           "dictionary": {
             "disabled": false,
             "onHeap": true,
             "useVarLengthDictionary": true
           }
         },
         "tierOverwrites": null
       }
     ],
     "ingestionConfig": {
       "transformConfigs": [],
       "continueOnError": false,
       "rowTimeValueCheck": false,
       "segmentTimeValueCheck": true
     },
     "isDimTable": false
   }
   ```
   
   - Based on this config, we expect the dictionary for C_NAME column to be loaded on-heap. However, that doesn't happen. 
   
   
   ### RCA
   - The issue is that `IndexLoadingConfig`  loads the configs for each column by calling  `createIndexConfigsByColName(tableConfig, schema, this::getDeserializer);` at [link](https://github.com/apache/pinot/blob/7132a2203f13478f450cbf8f0524ba72bdc267b7/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexLoadingConfig.java#L288) which uses the deserializer as defined [here](https://github.com/apache/pinot/blob/7132a2203f13478f450cbf8f0524ba72bdc267b7/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexLoadingConfig.java#L291).
   - This deserializer provider (onConflict.PICK_FIRST). gives priority to  `DictionaryIndexType.fromIndexLoadingConfig()` over `fieldConfigs`. As `DictionaryIndexType.fromIndexLoadingConfig()` returns a DictionaryConfig with onHeap set to false. So the fieldConfig settings are never considered. 
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Bugfix to load dictionary onHeap when defined in fieldConfigList [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #12265:
URL: https://github.com/apache/pinot/pull/12265#issuecomment-4190861870

   Close stale PR, please create a new one if the bug is still there.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Bugfix to load dictionary onHeap when defined in fieldConfigList [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #12265:
URL: https://github.com/apache/pinot/pull/12265#issuecomment-1959085779

   > Also, I don't see the CI running for this PR. Can you re-push the commit to trigger it?
   
   Nice catch. @vvivekiyer, can you do that?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Bugfix to load dictionary onHeap when defined in fieldConfigList [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12265:
URL: https://github.com/apache/pinot/pull/12265#discussion_r1498929879


##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/AbstractSerdeIndexContract.java:
##########
@@ -103,6 +106,14 @@ protected <C extends IndexConfig> C getActualConfig(String column, IndexType<C,
     return confMap.get(column).getConfig(type);
   }
 
+  protected <C extends IndexConfig> C getActualFromIndexLoadingConfig(String column, IndexType<C, ?, ?> type) {
+    IndexLoadingConfig indexLoadingConfig = new IndexLoadingConfig(_tableConfig, _schema);
+    Map<String, FieldIndexConfigs> confMap =
+        FieldIndexConfigsUtil.createIndexConfigsByColName(_tableConfig, _schema, indexLoadingConfig::getDeserializer);
+
+    return confMap.get(column).getConfig(type);
+  }

Review Comment:
   This can be changed with:
   ```java
     protected <C extends IndexConfig> C getActualFromIndexLoadingConfig(String column, IndexType<C, ?, ?> type) {
       IndexLoadingConfig indexLoadingConfig = new IndexLoadingConfig(_tableConfig, _schema);
       return indexLoadingConfig.calculateIndexConfigsByColName()
           .get(column)
           .getConfig(type);
     }
   ```
   
   By doing that we don't need to make `IndexingLoadingConfig.getDeserializer` public, so the encapsulation is better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Bugfix to load dictionary onHeap when defined in fieldConfigList [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12265:
URL: https://github.com/apache/pinot/pull/12265#discussion_r1498970724


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexLoadingConfig.java:
##########
@@ -303,7 +303,7 @@ private <C extends IndexConfig> ColumnConfigDeserializer<C> getDeserializer(Inde
         deserializer = IndexConfigDeserializer.fromMap(table -> fromIndexLoadingConfig);
       } else if (_segmentTier == null) {
         deserializer =
-            IndexConfigDeserializer.fromMap(table -> fromIndexLoadingConfig).withFallbackAlternative(stdDeserializer);
+            stdDeserializer.withFallbackAlternative(IndexConfigDeserializer.fromMap(table -> fromIndexLoadingConfig));

Review Comment:
   I think this is the change @Jackie-Jiang was worried about. And I think it makes sense to apply the change. Before this PR, our index config test verified the behavior in the ideal scenario where `IndexingLoadingConfig` was not used. But this PR adds a new check to verify that semantics are conserved when `IndexLoadingConfig` is used... and it proves there are issues right now in this case.
   
   I'm not 100% sure if this is the correct solution, but in indexes like ForwardIndex that are enabled by default, if we prioritizes `IndexConfigDeserializer.fromMap(table -> fromIndexLoadingConfig)` instead of `stdDeserializer`, the `stdDeserializer` is always ignored which is clearly incorrect.
   
   IMHO the correct solution would be to remove `IndexingLoadingConfig`, as we planned during `index-spi`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Bugfix to load dictionary onHeap when defined in fieldConfigList [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12265:
URL: https://github.com/apache/pinot/pull/12265#discussion_r1498926674


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexLoadingConfig.java:
##########
@@ -288,7 +288,7 @@ private Map<String, FieldIndexConfigs> calculateIndexConfigsByColName(@Nullable
     return FieldIndexConfigsUtil.createIndexConfigsByColName(tableConfig, schema, this::getDeserializer);
   }
 
-  private <C extends IndexConfig> ColumnConfigDeserializer<C> getDeserializer(IndexType<C, ?, ?> indexType) {
+  public <C extends IndexConfig> ColumnConfigDeserializer<C> getDeserializer(IndexType<C, ?, ?> indexType) {

Review Comment:
   I don't think we need to make this public. See the comment in AbstractSerderIndexContract



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Bugfix to load dictionary onHeap when defined in fieldConfigList [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 closed pull request #12265: Bugfix to load dictionary onHeap when defined in fieldConfigList
URL: https://github.com/apache/pinot/pull/12265


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Bugfix to load dictionary onHeap when defined in fieldConfigList [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #12265:
URL: https://github.com/apache/pinot/pull/12265#issuecomment-1889928243

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12265?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: `1 lines` in your changes are missing coverage. Please review.
   > Comparison is base [(`8f5fa80`)](https://app.codecov.io/gh/apache/pinot/commit/8f5fa804cd6e6fc9c029815488589cca370e7aea?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.50% compared to head [(`62cb349`)](https://app.codecov.io/gh/apache/pinot/pull/12265?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 34.67%.
   > Report is 15 commits behind head on master.
   
   | [Files](https://app.codecov.io/gh/apache/pinot/pull/12265?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...local/segment/index/loader/IndexLoadingConfig.java](https://app.codecov.io/gh/apache/pinot/pull/12265?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleExvYWRpbmdDb25maWcuamF2YQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12265?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #12265       +/-   ##
   =============================================
   - Coverage     61.50%   34.67%   -26.83%     
   - Complexity      207      946      +739     
   =============================================
     Files          2416     2340       -76     
     Lines        131179   127463     -3716     
     Branches      20246    19694      -552     
   =============================================
   - Hits          80686    44204    -36482     
   - Misses        44595    80108    +35513     
   + Partials       5898     3151     -2747     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12265/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/12265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-0.01%)` | :arrow_down: |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.67% <0.00%> (-26.71%)` | :arrow_down: |
   | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.41% <0.00%> (-15.09%)` | :arrow_down: |
   | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.66% <0.00%> (-26.70%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/12265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.67% <0.00%> (-26.83%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/12265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.43% <0.00%> (-15.07%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.43% <0.00%> (-0.17%)` | :arrow_down: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12265/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12265?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Bugfix to load dictionary onHeap when defined in fieldConfigList [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #12265:
URL: https://github.com/apache/pinot/pull/12265#discussion_r1498929879


##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/AbstractSerdeIndexContract.java:
##########
@@ -103,6 +106,14 @@ protected <C extends IndexConfig> C getActualConfig(String column, IndexType<C,
     return confMap.get(column).getConfig(type);
   }
 
+  protected <C extends IndexConfig> C getActualFromIndexLoadingConfig(String column, IndexType<C, ?, ?> type) {
+    IndexLoadingConfig indexLoadingConfig = new IndexLoadingConfig(_tableConfig, _schema);
+    Map<String, FieldIndexConfigs> confMap =
+        FieldIndexConfigsUtil.createIndexConfigsByColName(_tableConfig, _schema, indexLoadingConfig::getDeserializer);
+
+    return confMap.get(column).getConfig(type);
+  }

Review Comment:
   This can be changed to:
   ```java
     protected <C extends IndexConfig> C getActualFromIndexLoadingConfig(String column, IndexType<C, ?, ?> type) {
       IndexLoadingConfig indexLoadingConfig = new IndexLoadingConfig(_tableConfig, _schema);
       return indexLoadingConfig.calculateIndexConfigsByColName()
           .get(column)
           .getConfig(type);
     }
   ```
   
   By doing that we don't need to make `IndexingLoadingConfig.getDeserializer` public, so the encapsulation is better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Bugfix to load dictionary onHeap when defined in fieldConfigList [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #12265:
URL: https://github.com/apache/pinot/pull/12265#issuecomment-1957824938

   With this change, we will always prioritize the config from field config over indexing config (not only apply to dictionary). I think this is the correct behavior because field config always gives finer control, but this is also behavior change. @vvivekiyer Can you please modify the PR title and description to include these info?
   @gortiz Do you think this is the desired behavior?
   
   Also, I don't see the CI running for this PR. Can you re-push the commit to trigger it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Bugfix to load dictionary onHeap when defined in fieldConfigList [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #12265:
URL: https://github.com/apache/pinot/pull/12265#issuecomment-1895521145

   Can we restart pinot compatibility tests? We would need some committer to do so or @vvivekiyer you can push an empty commit to fool GHA


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org