You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/05/26 17:26:13 UTC
[GitHub] [pinot] geeknarrator opened a new pull request, #8781: Fix Upsert config validation to check for metrics aggregation
geeknarrator opened a new pull request, #8781:
URL: https://github.com/apache/pinot/pull/8781
Bugfix for: https://github.com/apache/pinot/issues/8226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] geeknarrator commented on a diff in pull request #8781: Fix Upsert config validation to check for metrics aggregation
Posted by GitBox <gi...@apache.org>.
geeknarrator commented on code in PR #8781:
URL: https://github.com/apache/pinot/pull/8781#discussion_r886350220
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -521,19 +520,32 @@ static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
Preconditions.checkState(streamConfig.hasLowLevelConsumerType() && !streamConfig.hasHighLevelConsumerType(),
"Upsert table must use low-level streaming consumer type");
// replica group is configured for routing
- Preconditions.checkState(
- tableConfig.getRoutingConfig() != null && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE
- .equalsIgnoreCase(tableConfig.getRoutingConfig().getInstanceSelectorType()),
+ Preconditions.checkState(tableConfig.getRoutingConfig() != null
+ && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE.equalsIgnoreCase(
+ tableConfig.getRoutingConfig().getInstanceSelectorType()),
"Upsert table must use strict replica-group (i.e. strictReplicaGroup) based routing");
// no startree index
- Preconditions.checkState(
- CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs()) && !tableConfig
- .getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
+ Preconditions.checkState(CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs())
+ && !tableConfig.getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
// comparison column exists
if (tableConfig.getUpsertConfig().getComparisonColumn() != null) {
String comparisonCol = tableConfig.getUpsertConfig().getComparisonColumn();
Preconditions.checkState(schema.hasColumn(comparisonCol), "The comparison column does not exist on schema");
}
+ Preconditions.checkState(!isAggregateMetricsEnabled(tableConfig),
+ "Metrics aggregation and upsert cannot be enabled together");
+ }
+
+ /**
+ * Checks if Metrics aggregation is enabled.
+ */
+ @VisibleForTesting
+ static boolean isAggregateMetricsEnabled(TableConfig config) {
+ boolean isAggregateMetricsEnabledInIndexingConfig =
+ Optional.ofNullable(config.getIndexingConfig()).map(IndexingConfig::isAggregateMetrics).orElse(false);
+ boolean hasAggregationConfigs = Optional.ofNullable(config.getIngestionConfig())
+ .map(ingestionConfig -> CollectionUtils.isNotEmpty(ingestionConfig.getAggregationConfigs())).orElse(false);
+ return isAggregateMetricsEnabledInIndexingConfig || hasAggregationConfigs;
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8781: Fix Upsert config validation to check for metrics aggregation
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #8781:
URL: https://github.com/apache/pinot/pull/8781#discussion_r883174305
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -534,6 +534,19 @@ static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
String comparisonCol = tableConfig.getUpsertConfig().getComparisonColumn();
Preconditions.checkState(schema.hasColumn(comparisonCol), "The comparison column does not exist on schema");
}
+ }
+ Preconditions.checkState(isAggregateMetricsEnabled(tableConfig),
+ "Metrics aggregation and upsert cannot be enabled together");
+ }
+
+ /**
+ * Checks if Metrics aggregation is enabled.
+ */
+ @VisibleForTesting
+ static boolean isAggregateMetricsEnabled(TableConfig config) {
+ if(config.getIndexingConfig() == null || config.getIngestionConfig() == null) {return false;
+ }
+ return config.getIndexingConfig().isAggregateMetrics() && CollectionUtils.isEmpty(config.getIngestionConfig().getAggregationConfigs());
Review Comment:
This should be `||` instead of `&&`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] geeknarrator commented on pull request #8781: Fix Upsert config validation to check for metrics aggregation
Posted by GitBox <gi...@apache.org>.
geeknarrator commented on PR #8781:
URL: https://github.com/apache/pinot/pull/8781#issuecomment-1140296153
> I don't think it can compile now because of the extra back brace. Can you please also fix the code format using [Pinot Style](https://docs.pinot.apache.org/developers/developers-and-contributors/code-setup#setup-ide)?
Thanks, I configured Pinot style.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] geeknarrator commented on pull request #8781: Fix Upsert config validation to check for metrics aggregation
Posted by GitBox <gi...@apache.org>.
geeknarrator commented on PR #8781:
URL: https://github.com/apache/pinot/pull/8781#issuecomment-1143158820
@Jackie-Jiang - Thanks for reviewing the changes :) I have addressed the remaining comments. Once you approve those, can I get it merged? I don't seem to have the access.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8781: Fix Upsert config validation to check for metrics aggregation
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #8781:
URL: https://github.com/apache/pinot/pull/8781#discussion_r886043582
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -521,19 +520,32 @@ static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
Preconditions.checkState(streamConfig.hasLowLevelConsumerType() && !streamConfig.hasHighLevelConsumerType(),
"Upsert table must use low-level streaming consumer type");
// replica group is configured for routing
- Preconditions.checkState(
- tableConfig.getRoutingConfig() != null && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE
- .equalsIgnoreCase(tableConfig.getRoutingConfig().getInstanceSelectorType()),
+ Preconditions.checkState(tableConfig.getRoutingConfig() != null
+ && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE.equalsIgnoreCase(
+ tableConfig.getRoutingConfig().getInstanceSelectorType()),
"Upsert table must use strict replica-group (i.e. strictReplicaGroup) based routing");
// no startree index
- Preconditions.checkState(
- CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs()) && !tableConfig
- .getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
+ Preconditions.checkState(CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs())
+ && !tableConfig.getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
// comparison column exists
if (tableConfig.getUpsertConfig().getComparisonColumn() != null) {
String comparisonCol = tableConfig.getUpsertConfig().getComparisonColumn();
Preconditions.checkState(schema.hasColumn(comparisonCol), "The comparison column does not exist on schema");
}
+ Preconditions.checkState(!isAggregateMetricsEnabled(tableConfig),
+ "Metrics aggregation and upsert cannot be enabled together");
+ }
+
+ /**
+ * Checks if Metrics aggregation is enabled.
+ */
+ @VisibleForTesting
+ static boolean isAggregateMetricsEnabled(TableConfig config) {
+ boolean isAggregateMetricsEnabledInIndexingConfig =
+ Optional.ofNullable(config.getIndexingConfig()).map(IndexingConfig::isAggregateMetrics).orElse(false);
+ boolean hasAggregationConfigs = Optional.ofNullable(config.getIngestionConfig())
+ .map(ingestionConfig -> CollectionUtils.isNotEmpty(ingestionConfig.getAggregationConfigs())).orElse(false);
+ return isAggregateMetricsEnabledInIndexingConfig || hasAggregationConfigs;
Review Comment:
Let's validate both configs cannot be enabled at the same time
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -521,19 +520,32 @@ static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
Preconditions.checkState(streamConfig.hasLowLevelConsumerType() && !streamConfig.hasHighLevelConsumerType(),
"Upsert table must use low-level streaming consumer type");
// replica group is configured for routing
- Preconditions.checkState(
- tableConfig.getRoutingConfig() != null && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE
- .equalsIgnoreCase(tableConfig.getRoutingConfig().getInstanceSelectorType()),
+ Preconditions.checkState(tableConfig.getRoutingConfig() != null
+ && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE.equalsIgnoreCase(
+ tableConfig.getRoutingConfig().getInstanceSelectorType()),
"Upsert table must use strict replica-group (i.e. strictReplicaGroup) based routing");
// no startree index
- Preconditions.checkState(
- CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs()) && !tableConfig
- .getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
+ Preconditions.checkState(CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs())
+ && !tableConfig.getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
// comparison column exists
if (tableConfig.getUpsertConfig().getComparisonColumn() != null) {
String comparisonCol = tableConfig.getUpsertConfig().getComparisonColumn();
Preconditions.checkState(schema.hasColumn(comparisonCol), "The comparison column does not exist on schema");
}
+ Preconditions.checkState(!isAggregateMetricsEnabled(tableConfig),
+ "Metrics aggregation and upsert cannot be enabled together");
+ }
+
+ /**
+ * Checks if Metrics aggregation is enabled.
+ */
+ @VisibleForTesting
+ static boolean isAggregateMetricsEnabled(TableConfig config) {
+ boolean isAggregateMetricsEnabledInIndexingConfig =
+ Optional.ofNullable(config.getIndexingConfig()).map(IndexingConfig::isAggregateMetrics).orElse(false);
+ boolean hasAggregationConfigs = Optional.ofNullable(config.getIngestionConfig())
+ .map(ingestionConfig -> CollectionUtils.isNotEmpty(ingestionConfig.getAggregationConfigs())).orElse(false);
Review Comment:
Suggest using the non-functional apis which performs better and more readable
```suggestion
boolean isAggregateMetricsEnabledInIndexingConfig = config.getIndexingConfig().isAggregateMetrics();
boolean hasAggregationConfigs = config.getIngestionConfig() != null && CollectionUtils.isNotEmpty(config.getIngestionConfig().getAggregationConfigs());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #8781: Fix Upsert config validation to check for metrics aggregation
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged PR #8781:
URL: https://github.com/apache/pinot/pull/8781
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #8781: Fix Upsert config validation to check for metrics aggregation
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #8781:
URL: https://github.com/apache/pinot/pull/8781#issuecomment-1144043889
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8781?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8781](https://codecov.io/gh/apache/pinot/pull/8781?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bc5b6d2) into [master](https://codecov.io/gh/apache/pinot/commit/73316a6cf7016bf028b002a93b27da1391ce88dd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (73316a6) will **decrease** coverage by `42.96%`.
> The diff coverage is `61.23%`.
> :exclamation: Current head bc5b6d2 differs from pull request most recent head 7e7e717. Consider uploading reports for the commit 7e7e717 to get more accurate results
```diff
@@ Coverage Diff @@
## master #8781 +/- ##
=============================================
- Coverage 68.51% 25.54% -42.97%
+ Complexity 4601 45 -4556
=============================================
Files 1736 1723 -13
Lines 91196 90966 -230
Branches 13633 13610 -23
=============================================
- Hits 62480 23240 -39240
- Misses 24330 65505 +41175
+ Partials 4386 2221 -2165
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `25.54% <61.23%> (?)` | |
| unittests1 | `?` | |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8781?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...roker/requesthandler/BaseBrokerRequestHandler.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvQmFzZUJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `61.59% <0.00%> (-5.23%)` | :arrow_down: |
| [...uting/segmentselector/RealtimeSegmentSelector.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50c2VsZWN0b3IvUmVhbHRpbWVTZWdtZW50U2VsZWN0b3IuamF2YQ==) | `88.13% <0.00%> (-8.69%)` | :arrow_down: |
| [...he/pinot/common/tier/FixedTierSegmentSelector.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdGllci9GaXhlZFRpZXJTZWdtZW50U2VsZWN0b3IuamF2YQ==) | `0.00% <0.00%> (-93.34%)` | :arrow_down: |
| [.../helix/core/realtime/SegmentCompletionManager.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1NlZ21lbnRDb21wbGV0aW9uTWFuYWdlci5qYXZh) | `37.47% <0.00%> (-34.54%)` | :arrow_down: |
| [.../org/apache/pinot/core/common/MinionConstants.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vTWluaW9uQ29uc3RhbnRzLmphdmE=) | `0.00% <ø> (ø)` | |
| [...t/core/plan/AggregationGroupByOrderByPlanNode.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uR3JvdXBCeU9yZGVyQnlQbGFuTm9kZS5qYXZh) | `45.45% <0.00%> (-36.37%)` | :arrow_down: |
| [...ot/plugin/stream/kafka20/RowMetadataExtractor.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1zdHJlYW0taW5nZXN0aW9uL3Bpbm90LWthZmthLTIuMC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcGx1Z2luL3N0cmVhbS9rYWZrYTIwL1Jvd01ldGFkYXRhRXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...l/indexsegment/immutable/ImmutableSegmentImpl.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRJbXBsLmphdmE=) | `0.00% <0.00%> (-65.63%)` | :arrow_down: |
| [...eaderwriter/RealtimeIndexOffHeapMemoryManager.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pby9yZWFkZXJ3cml0ZXIvUmVhbHRpbWVJbmRleE9mZkhlYXBNZW1vcnlNYW5hZ2VyLmphdmE=) | `0.00% <0.00%> (-80.65%)` | :arrow_down: |
| [...dex/readers/bloom/GuavaBloomFilterReaderUtils.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvYmxvb20vR3VhdmFCbG9vbUZpbHRlclJlYWRlclV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [1344 more](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8781?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8781?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [73316a6...7e7e717](https://codecov.io/gh/apache/pinot/pull/8781?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] geeknarrator commented on a diff in pull request #8781: Fix Upsert config validation to check for metrics aggregation
Posted by GitBox <gi...@apache.org>.
geeknarrator commented on code in PR #8781:
URL: https://github.com/apache/pinot/pull/8781#discussion_r886342049
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -521,19 +520,32 @@ static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
Preconditions.checkState(streamConfig.hasLowLevelConsumerType() && !streamConfig.hasHighLevelConsumerType(),
"Upsert table must use low-level streaming consumer type");
// replica group is configured for routing
- Preconditions.checkState(
- tableConfig.getRoutingConfig() != null && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE
- .equalsIgnoreCase(tableConfig.getRoutingConfig().getInstanceSelectorType()),
+ Preconditions.checkState(tableConfig.getRoutingConfig() != null
+ && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE.equalsIgnoreCase(
+ tableConfig.getRoutingConfig().getInstanceSelectorType()),
"Upsert table must use strict replica-group (i.e. strictReplicaGroup) based routing");
// no startree index
- Preconditions.checkState(
- CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs()) && !tableConfig
- .getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
+ Preconditions.checkState(CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs())
+ && !tableConfig.getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
// comparison column exists
if (tableConfig.getUpsertConfig().getComparisonColumn() != null) {
String comparisonCol = tableConfig.getUpsertConfig().getComparisonColumn();
Preconditions.checkState(schema.hasColumn(comparisonCol), "The comparison column does not exist on schema");
}
+ Preconditions.checkState(!isAggregateMetricsEnabled(tableConfig),
+ "Metrics aggregation and upsert cannot be enabled together");
+ }
+
+ /**
+ * Checks if Metrics aggregation is enabled.
+ */
+ @VisibleForTesting
+ static boolean isAggregateMetricsEnabled(TableConfig config) {
+ boolean isAggregateMetricsEnabledInIndexingConfig =
+ Optional.ofNullable(config.getIndexingConfig()).map(IndexingConfig::isAggregateMetrics).orElse(false);
+ boolean hasAggregationConfigs = Optional.ofNullable(config.getIngestionConfig())
+ .map(ingestionConfig -> CollectionUtils.isNotEmpty(ingestionConfig.getAggregationConfigs())).orElse(false);
+ return isAggregateMetricsEnabledInIndexingConfig || hasAggregationConfigs;
Review Comment:
I see.. I didn't know that is a requirement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] geeknarrator commented on a diff in pull request #8781: Fix Upsert config validation to check for metrics aggregation
Posted by GitBox <gi...@apache.org>.
geeknarrator commented on code in PR #8781:
URL: https://github.com/apache/pinot/pull/8781#discussion_r884155819
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -534,6 +534,19 @@ static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
String comparisonCol = tableConfig.getUpsertConfig().getComparisonColumn();
Preconditions.checkState(schema.hasColumn(comparisonCol), "The comparison column does not exist on schema");
}
+ }
+ Preconditions.checkState(isAggregateMetricsEnabled(tableConfig),
+ "Metrics aggregation and upsert cannot be enabled together");
+ }
+
+ /**
+ * Checks if Metrics aggregation is enabled.
+ */
+ @VisibleForTesting
+ static boolean isAggregateMetricsEnabled(TableConfig config) {
+ if(config.getIndexingConfig() == null || config.getIngestionConfig() == null) {return false;
+ }
+ return config.getIndexingConfig().isAggregateMetrics() && CollectionUtils.isEmpty(config.getIngestionConfig().getAggregationConfigs());
Review Comment:
I think I referred `MutableSegmentImpl` and misunderstood. Fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] geeknarrator commented on pull request #8781: Fix Upsert config validation to check for metrics aggregation
Posted by GitBox <gi...@apache.org>.
geeknarrator commented on PR #8781:
URL: https://github.com/apache/pinot/pull/8781#issuecomment-1146176170
@Jackie-Jiang - Could you please help me merge this? I have resolved the conflict. (also the formatting changes are done by the pinot checkstyle configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] geeknarrator commented on a diff in pull request #8781: Fix Upsert config validation to check for metrics aggregation
Posted by GitBox <gi...@apache.org>.
geeknarrator commented on code in PR #8781:
URL: https://github.com/apache/pinot/pull/8781#discussion_r886341586
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -521,19 +520,32 @@ static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
Preconditions.checkState(streamConfig.hasLowLevelConsumerType() && !streamConfig.hasHighLevelConsumerType(),
"Upsert table must use low-level streaming consumer type");
// replica group is configured for routing
- Preconditions.checkState(
- tableConfig.getRoutingConfig() != null && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE
- .equalsIgnoreCase(tableConfig.getRoutingConfig().getInstanceSelectorType()),
+ Preconditions.checkState(tableConfig.getRoutingConfig() != null
+ && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE.equalsIgnoreCase(
+ tableConfig.getRoutingConfig().getInstanceSelectorType()),
"Upsert table must use strict replica-group (i.e. strictReplicaGroup) based routing");
// no startree index
- Preconditions.checkState(
- CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs()) && !tableConfig
- .getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
+ Preconditions.checkState(CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs())
+ && !tableConfig.getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
// comparison column exists
if (tableConfig.getUpsertConfig().getComparisonColumn() != null) {
String comparisonCol = tableConfig.getUpsertConfig().getComparisonColumn();
Preconditions.checkState(schema.hasColumn(comparisonCol), "The comparison column does not exist on schema");
}
+ Preconditions.checkState(!isAggregateMetricsEnabled(tableConfig),
+ "Metrics aggregation and upsert cannot be enabled together");
+ }
+
+ /**
+ * Checks if Metrics aggregation is enabled.
+ */
+ @VisibleForTesting
+ static boolean isAggregateMetricsEnabled(TableConfig config) {
+ boolean isAggregateMetricsEnabledInIndexingConfig =
+ Optional.ofNullable(config.getIndexingConfig()).map(IndexingConfig::isAggregateMetrics).orElse(false);
+ boolean hasAggregationConfigs = Optional.ofNullable(config.getIngestionConfig())
+ .map(ingestionConfig -> CollectionUtils.isNotEmpty(ingestionConfig.getAggregationConfigs())).orElse(false);
Review Comment:
👍 sgtm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org