You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2026/07/10 00:28:07 UTC
[PR] Refactor upsert/dedup config validation into per-mode branches [pinot]
Jackie-Jiang opened a new pull request, #18956:
URL: https://github.com/apache/pinot/pull/18956
## Summary
Restructures `TableConfigUtils#validateUpsertAndDedupConfig` to return early when neither upsert nor dedup is enabled, then validate the shared requirements (mutual exclusivity, primary keys, strict replica-group routing, no COMPLETED instance partitions, OFFLINE restrictions) followed by an upsert- or dedup-specific branch.
- TTL, MD5-disabled, and partial-upsert validation move into the enabled branch (each only runs for the applicable mode).
- The now-redundant `validateAggregateMetricsForUpsertConfig` is removed — the aggregation vs upsert/dedup incompatibility is validated in `validateMetricsAggregation`.
- Uses the non-deprecated `TableConfig#isUpsertEnabled()` / `#isDedupEnabled()` helpers instead of `getUpsertMode()`.
- Folds in related helper cleanups (`isCommitTimeCompactionEnabled`) and the accompanying test refactor.
## Dependency
Stacked on #18955 (the aggregation vs upsert/dedup incompatibility check now lives in `validateMetricsAggregation`, introduced there). Until #18955 merges, this PR's diff includes that commit; please review/merge after #18955.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Refactor upsert/dedup config validation into per-mode branches [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #18956:
URL: https://github.com/apache/pinot/pull/18956#discussion_r3555670508
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -443,325 +442,371 @@ private static boolean isValidPeerDownloadScheme(String peerSegmentDownloadSchem
|| CommonConstants.HTTPS_PROTOCOL.equalsIgnoreCase(peerSegmentDownloadScheme);
}
- /**
- * Validates the following:
- * 1. validity of filter function
- * 2. checks for duplicate transform configs
- * 3. checks for null column name or transform function in transform config
- * 4. validity of transform function string
- * 5. checks for source fields used in destination columns
- * 6. ingestion type for dimension tables
- */
+ /// Validates the table's [IngestionConfig] together with the closely related metrics-aggregation config, covering:
+ /// - Metrics aggregation (both the `aggregateMetrics` flag and ingestion `aggregationConfigs`), delegated to
+ /// [#validateMetricsAggregation].
+ /// - Batch ingestion: each batch config map is well-formed, and a dimension table has batch ingestion configured
+ /// with `REFRESH` segment ingestion type.
+ /// - Stream ingestion: streams are declared in only one place, at least one stream is present, and pauseless
+ /// consumption has a valid `peerSegmentDownloadScheme`.
+ /// - Filter config: the filter function is valid and not a disabled Groovy expression.
+ /// - Source field configs: no source field is duplicated within the same complex-type phase.
+ /// - Enrichment configs: each config is valid.
+ /// - Transform configs: non-null column and function, no duplicate destination, and each destination is a schema
+ /// column, an intermediate consumed by another transform, or an aggregation source column; the function is valid
+ /// and does not reference its own destination.
+ /// - Complex-type config: no schema field collides with a `prefixesToRename` prefix.
+ /// - Schema-conforming transformer config.
@VisibleForTesting
public static void validateIngestionConfig(TableConfig tableConfig, Schema schema) {
+ // All metrics-aggregation validation lives here; it returns the columns referenced as aggregation sources, which
+ // a transform config is allowed to target as its destination (see the transform validation below).
+ Set<String> aggregationSourceColumns = validateMetricsAggregation(tableConfig, schema);
+
IngestionConfig ingestionConfig = tableConfig.getIngestionConfig();
+ if (ingestionConfig == null) {
+ return;
+ }
Review Comment:
validateIngestionConfig() returns early when ingestionConfig is null, which makes the dimension-table requirement check below ("Dimension tables must have batch ingestion configuration") unreachable. As a result, an OFFLINE dimension table can pass validation without any batch ingestion config, contrary to the explicit Preconditions check later in this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Refactor upsert/dedup config validation into per-mode branches [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18956:
URL: https://github.com/apache/pinot/pull/18956#issuecomment-4931066252
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18956?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `14.34783%` with `197 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 56.96%. Comparing base ([`f29b359`](https://app.codecov.io/gh/apache/pinot/commit/f29b359b8464b8d8f54aefa92e9fec37544f94e7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`c0b8194`](https://app.codecov.io/gh/apache/pinot/commit/c0b8194651647159e3550f0c4727cf2e2357207a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 4 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/18956?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...he/pinot/segment/local/utils/TableConfigUtils.java](https://app.codecov.io/gh/apache/pinot/pull/18956?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Futils%2FTableConfigUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9UYWJsZUNvbmZpZ1V0aWxzLmphdmE=) | 14.60% | [173 Missing and 20 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18956?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...local/indexsegment/mutable/MutableSegmentImpl.java](https://app.codecov.io/gh/apache/pinot/pull/18956?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Findexsegment%2Fmutable%2FMutableSegmentImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvbXV0YWJsZS9NdXRhYmxlU2VnbWVudEltcGwuamF2YQ==) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18956?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (f29b359) and HEAD (c0b8194). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (f29b359) | HEAD (c0b8194) |
>|------|------|------|
>|java-21|5|1|
>|unittests|2|1|
>|temurin|5|1|
>|integration|3|0|
>|integration1|1|0|
>|unittests2|1|0|
>|custom-integration1|1|0|
>|integration2|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18956 +/- ##
============================================
- Coverage 64.93% 56.96% -7.97%
+ Complexity 1347 1 -1346
============================================
Files 3395 2605 -790
Lines 212533 153745 -58788
Branches 33515 25075 -8440
============================================
- Hits 138012 87585 -50427
+ Misses 63379 58699 -4680
+ Partials 11142 7461 -3681
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/18956/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18956/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/18956/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/18956/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/18956/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/18956/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.96% <14.34%> (-7.97%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/18956/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.96% <14.34%> (-7.97%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/18956/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.96% <14.34%> (-7.97%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18956/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.96% <14.34%> (+<0.01%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18956/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/18956?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Refactor upsert/dedup config validation into per-mode branches [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #18956:
URL: https://github.com/apache/pinot/pull/18956
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Refactor upsert/dedup config validation into per-mode branches [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #18956:
URL: https://github.com/apache/pinot/pull/18956#discussion_r3556911095
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -443,325 +442,371 @@ private static boolean isValidPeerDownloadScheme(String peerSegmentDownloadSchem
|| CommonConstants.HTTPS_PROTOCOL.equalsIgnoreCase(peerSegmentDownloadScheme);
}
- /**
- * Validates the following:
- * 1. validity of filter function
- * 2. checks for duplicate transform configs
- * 3. checks for null column name or transform function in transform config
- * 4. validity of transform function string
- * 5. checks for source fields used in destination columns
- * 6. ingestion type for dimension tables
- */
+ /// Validates the table's [IngestionConfig] together with the closely related metrics-aggregation config, covering:
+ /// - Metrics aggregation (both the `aggregateMetrics` flag and ingestion `aggregationConfigs`), delegated to
+ /// [#validateMetricsAggregation].
+ /// - Batch ingestion: each batch config map is well-formed, and a dimension table has batch ingestion configured
+ /// with `REFRESH` segment ingestion type.
+ /// - Stream ingestion: streams are declared in only one place, at least one stream is present, and pauseless
+ /// consumption has a valid `peerSegmentDownloadScheme`.
+ /// - Filter config: the filter function is valid and not a disabled Groovy expression.
+ /// - Source field configs: no source field is duplicated within the same complex-type phase.
+ /// - Enrichment configs: each config is valid.
+ /// - Transform configs: non-null column and function, no duplicate destination, and each destination is a schema
+ /// column, an intermediate consumed by another transform, or an aggregation source column; the function is valid
+ /// and does not reference its own destination.
+ /// - Complex-type config: no schema field collides with a `prefixesToRename` prefix.
+ /// - Schema-conforming transformer config.
@VisibleForTesting
public static void validateIngestionConfig(TableConfig tableConfig, Schema schema) {
+ // All metrics-aggregation validation lives here; it returns the columns referenced as aggregation sources, which
+ // a transform config is allowed to target as its destination (see the transform validation below).
+ Set<String> aggregationSourceColumns = validateMetricsAggregation(tableConfig, schema);
+
IngestionConfig ingestionConfig = tableConfig.getIngestionConfig();
+ if (ingestionConfig == null) {
+ return;
+ }
Review Comment:
This is behavior-equivalent to `master`: the dim-table batch-config check previously lived inside `if (ingestionConfig != null) { ... }`, so it was equally skipped when `ingestionConfig` is null. The early-return doesn't change that, so it isn't a regression. The latent gap (a dimension table with no `ingestionConfig` at all) predates this change and is out of scope for this refactor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org