You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "deepthi912 (via GitHub)" <gi...@apache.org> on 2026/05/22 02:19:33 UTC

[PR] Scope overwriteTableConfigForTier Jackson roundtrip to sub-configs that can hold tier overrides [pinot]

deepthi912 opened a new pull request, #18563:
URL: https://github.com/apache/pinot/pull/18563

     **Problem**
   `overwriteTableConfigForTier` runs on every segment during checkSegmentsReload polls. The current code does a full Jackson roundtrip on the entire TableConfig — serializing IndexingConfig,
     FieldConfigList, ingestion, validation, tenant, quota, routing, all of it — even though tier overrides can only live on IndexingConfig._tierOverwrites and FieldConfig._tierOverwrites.
   
    **Worst case:** tables with no tierOverwrites defined anywhere would still pay the full serialize cost before, because the master code serializes upfront and only then checks if there's anything to override.
   
   overwriteTableConfigForTier allocated ~4.5 GB (319 calls × ~14 MB each). Largest single allocator on the reload-check path.
   
     Fix
   
     Only roundtrip the pieces that can have overrides:
     - If IndexingConfig.getTierOverwrites() has an entry for this tier → roundtrip just IndexingConfig. Otherwise return the original reference.
     - For each FieldConfig with an override → roundtrip just that FieldConfig. Otherwise keep the original reference.
     - If nothing changed → return the original TableConfig reference (no allocation).
     - If something changed → shallow-copy TableConfig and swap the two affected fields.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Scope overwriteTableConfigForTier Table config Jackson to sub-configs that can hold tier overrides [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #18563:
URL: https://github.com/apache/pinot/pull/18563


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Scope overwriteTableConfigForTier Table config Jackson to sub-configs that can hold tier overrides [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #18563:
URL: https://github.com/apache/pinot/pull/18563#discussion_r3290923149


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -2088,52 +2080,82 @@ public static TableConfig overwriteTableConfigForTier(TableConfig tableConfig, @
       return tableConfig;
     }
     try {
-      boolean updated = false;
-      JsonNode tblCfgJson = tableConfig.toJsonNode();
-      // Apply tier specific overwrites for `tableIndexConfig`
-      JsonNode tblIdxCfgJson = tblCfgJson.get(TableConfig.INDEXING_CONFIG_KEY);
-      if (tblIdxCfgJson != null && tblIdxCfgJson.has(TableConfig.TIER_OVERWRITES_KEY)) {
-        JsonNode tierCfgJson = tblIdxCfgJson.get(TableConfig.TIER_OVERWRITES_KEY).get(tier);
-        if (tierCfgJson != null) {
-          LOGGER.debug("Got table index config overwrites: {} for tier: {}", tierCfgJson, tier);
-          overwriteConfig(tblIdxCfgJson, tierCfgJson);
-          updated = true;
-        }
-      }
-      // Apply tier specific overwrites for `fieldConfigList`
-      JsonNode fieldCfgListJson = tblCfgJson.get(TableConfig.FIELD_CONFIG_LIST_KEY);
-      if (fieldCfgListJson != null && fieldCfgListJson.isArray()) {
-        Iterator<JsonNode> fieldCfgListItr = fieldCfgListJson.elements();
-        while (fieldCfgListItr.hasNext()) {
-          JsonNode fieldCfgJson = fieldCfgListItr.next();
-          if (!fieldCfgJson.has(TableConfig.TIER_OVERWRITES_KEY)) {
-            continue;
-          }
-          JsonNode tierCfgJson = fieldCfgJson.get(TableConfig.TIER_OVERWRITES_KEY).get(tier);
-          if (tierCfgJson != null) {
-            LOGGER.debug("Got field index config overwrites: {} for tier: {}", tierCfgJson, tier);
-            overwriteConfig(fieldCfgJson, tierCfgJson);
-            updated = true;
-          }
-        }
-      }
-      if (updated) {
-        LOGGER.debug("Got overwritten table config: {} for tier: {}", tblCfgJson, tier);
-        return JsonUtils.jsonNodeToObject(tblCfgJson, TableConfig.class);
-      } else {
-        LOGGER.debug("No table config overwrites for tier: {}", tier);
+      IndexingConfig effectiveIndexing = applyIndexingConfigTierOverride(tableConfig.getIndexingConfig(), tier);
+      List<FieldConfig> effectiveFields = applyFieldConfigListTierOverrides(tableConfig.getFieldConfigList(), tier);
+      if (effectiveIndexing == tableConfig.getIndexingConfig()
+          && effectiveFields == tableConfig.getFieldConfigList()) {
         return tableConfig;
       }
+      TableConfig overwritten = new TableConfig(tableConfig);
+      overwritten.setIndexingConfig(effectiveIndexing);
+      overwritten.setFieldConfigList(effectiveFields);
+      return overwritten;
     } catch (IOException e) {
       LOGGER.warn("Failed to overwrite table config for tier: {} for table: {}", tier, tableConfig.getTableName(), e);
       return tableConfig;
     }
   }
 
-  private static void overwriteConfig(JsonNode oldCfg, JsonNode newCfg) {
-    for (Map.Entry<String, JsonNode> cfgEntry : newCfg.properties()) {
-      ((ObjectNode) oldCfg).set(cfgEntry.getKey(), cfgEntry.getValue());
+  @Nullable
+  private static IndexingConfig applyIndexingConfigTierOverride(@Nullable IndexingConfig original, String tier)
+      throws IOException {
+    if (original == null) {
+      return null;
+    }
+    JsonNode tierOverwrites = original.getTierOverwrites();
+    if (tierOverwrites == null || !tierOverwrites.has(tier)) {
+      return original;
+    }
+    JsonNode override = tierOverwrites.get(tier);
+    if (!override.isObject()) {
+      return original;
+    }
+    ObjectNode merged = (ObjectNode) JsonUtils.objectToJsonNode(original);
+    for (Map.Entry<String, JsonNode> entry : override.properties()) {
+      merged.set(entry.getKey(), entry.getValue());
+    }
+    return JsonUtils.jsonNodeToObject(merged, IndexingConfig.class);
+  }
+
+  @Nullable
+  private static List<FieldConfig> applyFieldConfigListTierOverrides(@Nullable List<FieldConfig> original, String tier)
+      throws IOException {
+    if (original == null || original.isEmpty()) {

Review Comment:
   (nit) CollectionUtils.isEmpty()`



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -2088,52 +2080,82 @@ public static TableConfig overwriteTableConfigForTier(TableConfig tableConfig, @
       return tableConfig;
     }
     try {
-      boolean updated = false;
-      JsonNode tblCfgJson = tableConfig.toJsonNode();
-      // Apply tier specific overwrites for `tableIndexConfig`
-      JsonNode tblIdxCfgJson = tblCfgJson.get(TableConfig.INDEXING_CONFIG_KEY);
-      if (tblIdxCfgJson != null && tblIdxCfgJson.has(TableConfig.TIER_OVERWRITES_KEY)) {
-        JsonNode tierCfgJson = tblIdxCfgJson.get(TableConfig.TIER_OVERWRITES_KEY).get(tier);
-        if (tierCfgJson != null) {
-          LOGGER.debug("Got table index config overwrites: {} for tier: {}", tierCfgJson, tier);
-          overwriteConfig(tblIdxCfgJson, tierCfgJson);
-          updated = true;
-        }
-      }
-      // Apply tier specific overwrites for `fieldConfigList`
-      JsonNode fieldCfgListJson = tblCfgJson.get(TableConfig.FIELD_CONFIG_LIST_KEY);
-      if (fieldCfgListJson != null && fieldCfgListJson.isArray()) {
-        Iterator<JsonNode> fieldCfgListItr = fieldCfgListJson.elements();
-        while (fieldCfgListItr.hasNext()) {
-          JsonNode fieldCfgJson = fieldCfgListItr.next();
-          if (!fieldCfgJson.has(TableConfig.TIER_OVERWRITES_KEY)) {
-            continue;
-          }
-          JsonNode tierCfgJson = fieldCfgJson.get(TableConfig.TIER_OVERWRITES_KEY).get(tier);
-          if (tierCfgJson != null) {
-            LOGGER.debug("Got field index config overwrites: {} for tier: {}", tierCfgJson, tier);
-            overwriteConfig(fieldCfgJson, tierCfgJson);
-            updated = true;
-          }
-        }
-      }
-      if (updated) {
-        LOGGER.debug("Got overwritten table config: {} for tier: {}", tblCfgJson, tier);
-        return JsonUtils.jsonNodeToObject(tblCfgJson, TableConfig.class);
-      } else {
-        LOGGER.debug("No table config overwrites for tier: {}", tier);
+      IndexingConfig effectiveIndexing = applyIndexingConfigTierOverride(tableConfig.getIndexingConfig(), tier);
+      List<FieldConfig> effectiveFields = applyFieldConfigListTierOverrides(tableConfig.getFieldConfigList(), tier);
+      if (effectiveIndexing == tableConfig.getIndexingConfig()
+          && effectiveFields == tableConfig.getFieldConfigList()) {
         return tableConfig;
       }
+      TableConfig overwritten = new TableConfig(tableConfig);
+      overwritten.setIndexingConfig(effectiveIndexing);
+      overwritten.setFieldConfigList(effectiveFields);
+      return overwritten;
     } catch (IOException e) {
       LOGGER.warn("Failed to overwrite table config for tier: {} for table: {}", tier, tableConfig.getTableName(), e);
       return tableConfig;
     }
   }
 
-  private static void overwriteConfig(JsonNode oldCfg, JsonNode newCfg) {
-    for (Map.Entry<String, JsonNode> cfgEntry : newCfg.properties()) {
-      ((ObjectNode) oldCfg).set(cfgEntry.getKey(), cfgEntry.getValue());
+  @Nullable
+  private static IndexingConfig applyIndexingConfigTierOverride(@Nullable IndexingConfig original, String tier)

Review Comment:
   (nit) `IndexingConfig` always exists in table config



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -2088,52 +2080,82 @@ public static TableConfig overwriteTableConfigForTier(TableConfig tableConfig, @
       return tableConfig;
     }
     try {
-      boolean updated = false;
-      JsonNode tblCfgJson = tableConfig.toJsonNode();
-      // Apply tier specific overwrites for `tableIndexConfig`
-      JsonNode tblIdxCfgJson = tblCfgJson.get(TableConfig.INDEXING_CONFIG_KEY);
-      if (tblIdxCfgJson != null && tblIdxCfgJson.has(TableConfig.TIER_OVERWRITES_KEY)) {
-        JsonNode tierCfgJson = tblIdxCfgJson.get(TableConfig.TIER_OVERWRITES_KEY).get(tier);
-        if (tierCfgJson != null) {
-          LOGGER.debug("Got table index config overwrites: {} for tier: {}", tierCfgJson, tier);
-          overwriteConfig(tblIdxCfgJson, tierCfgJson);
-          updated = true;
-        }
-      }
-      // Apply tier specific overwrites for `fieldConfigList`
-      JsonNode fieldCfgListJson = tblCfgJson.get(TableConfig.FIELD_CONFIG_LIST_KEY);
-      if (fieldCfgListJson != null && fieldCfgListJson.isArray()) {
-        Iterator<JsonNode> fieldCfgListItr = fieldCfgListJson.elements();
-        while (fieldCfgListItr.hasNext()) {
-          JsonNode fieldCfgJson = fieldCfgListItr.next();
-          if (!fieldCfgJson.has(TableConfig.TIER_OVERWRITES_KEY)) {
-            continue;
-          }
-          JsonNode tierCfgJson = fieldCfgJson.get(TableConfig.TIER_OVERWRITES_KEY).get(tier);
-          if (tierCfgJson != null) {
-            LOGGER.debug("Got field index config overwrites: {} for tier: {}", tierCfgJson, tier);
-            overwriteConfig(fieldCfgJson, tierCfgJson);
-            updated = true;
-          }
-        }
-      }
-      if (updated) {
-        LOGGER.debug("Got overwritten table config: {} for tier: {}", tblCfgJson, tier);
-        return JsonUtils.jsonNodeToObject(tblCfgJson, TableConfig.class);
-      } else {
-        LOGGER.debug("No table config overwrites for tier: {}", tier);
+      IndexingConfig effectiveIndexing = applyIndexingConfigTierOverride(tableConfig.getIndexingConfig(), tier);
+      List<FieldConfig> effectiveFields = applyFieldConfigListTierOverrides(tableConfig.getFieldConfigList(), tier);
+      if (effectiveIndexing == tableConfig.getIndexingConfig()
+          && effectiveFields == tableConfig.getFieldConfigList()) {
         return tableConfig;
       }
+      TableConfig overwritten = new TableConfig(tableConfig);
+      overwritten.setIndexingConfig(effectiveIndexing);
+      overwritten.setFieldConfigList(effectiveFields);
+      return overwritten;
     } catch (IOException e) {
       LOGGER.warn("Failed to overwrite table config for tier: {} for table: {}", tier, tableConfig.getTableName(), e);
       return tableConfig;
     }
   }
 
-  private static void overwriteConfig(JsonNode oldCfg, JsonNode newCfg) {
-    for (Map.Entry<String, JsonNode> cfgEntry : newCfg.properties()) {
-      ((ObjectNode) oldCfg).set(cfgEntry.getKey(), cfgEntry.getValue());
+  @Nullable
+  private static IndexingConfig applyIndexingConfigTierOverride(@Nullable IndexingConfig original, String tier)
+      throws IOException {
+    if (original == null) {
+      return null;
+    }
+    JsonNode tierOverwrites = original.getTierOverwrites();
+    if (tierOverwrites == null || !tierOverwrites.has(tier)) {
+      return original;
+    }
+    JsonNode override = tierOverwrites.get(tier);
+    if (!override.isObject()) {
+      return original;
+    }
+    ObjectNode merged = (ObjectNode) JsonUtils.objectToJsonNode(original);
+    for (Map.Entry<String, JsonNode> entry : override.properties()) {
+      merged.set(entry.getKey(), entry.getValue());
+    }
+    return JsonUtils.jsonNodeToObject(merged, IndexingConfig.class);
+  }
+
+  @Nullable
+  private static List<FieldConfig> applyFieldConfigListTierOverrides(@Nullable List<FieldConfig> original, String tier)
+      throws IOException {
+    if (original == null || original.isEmpty()) {
+      return original;
+    }
+    List<FieldConfig> result = null;
+    for (int i = 0; i < original.size(); i++) {
+      FieldConfig config = original.get(i);
+      FieldConfig effective = applyFieldConfigTierOverride(config, tier);
+      if (effective != config) {
+        if (result == null) {
+          result = new ArrayList<>(original);
+        }
+        result.set(i, effective);
+      }
+    }
+    return result != null ? result : original;
+  }
+
+  @Nullable
+  private static FieldConfig applyFieldConfigTierOverride(@Nullable FieldConfig original, String tier)

Review Comment:
   (nit) original can never be `null`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Scope overwriteTableConfigForTier Table config Jackson to sub-configs that can hold tier overrides [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18563:
URL: https://github.com/apache/pinot/pull/18563#issuecomment-4514725374

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18563?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `13.15789%` with `33 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 56.73%. Comparing base ([`7358d5a`](https://app.codecov.io/gh/apache/pinot/commit/7358d5a694a23d6e13bfe6c266885b7cd294d5d9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`44f7ad1`](https://app.codecov.io/gh/apache/pinot/commit/44f7ad1902bc418c76aca058a092192b22b27710?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 4 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/18563?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...he/pinot/segment/local/utils/TableConfigUtils.java](https://app.codecov.io/gh/apache/pinot/pull/18563?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Futils%2FTableConfigUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9UYWJsZUNvbmZpZ1V0aWxzLmphdmE=) | 13.15% | [28 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18563?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   > :exclamation:  There is a different number of reports uploaded between BASE (7358d5a) and HEAD (44f7ad1). Click for more details.
   > 
   > <details><summary>HEAD has 16 uploads less than BASE</summary>
   >
   >| Flag | BASE (7358d5a) | HEAD (44f7ad1) |
   >|------|------|------|
   >|java-21|5|1|
   >|unittests|2|1|
   >|temurin|5|1|
   >|integration|3|0|
   >|integration1|1|0|
   >|unittests2|1|0|
   >|custom-integration1|1|0|
   >|integration2|1|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #18563      +/-   ##
   ============================================
   - Coverage     64.26%   56.73%   -7.54%     
   ============================================
     Files          3311     2561     -750     
     Lines        203829   148643   -55186     
     Branches      31722    24033    -7689     
   ============================================
   - Hits         130989    84331   -46658     
   + Misses        62326    57140    -5186     
   + Partials      10514     7172    -3342     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/18563/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18563/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/18563/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/18563/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/18563/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/18563/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.73% <13.15%> (-7.54%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/18563/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.73% <13.15%> (-7.54%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/18563/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.73% <13.15%> (-7.53%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18563/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.73% <13.15%> (+0.01%)` | :arrow_up: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18563/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18563?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org