You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/05/26 17:26:13 UTC

[GitHub] [pinot] geeknarrator opened a new pull request, #8781: Fix Upsert config validation to check for metrics aggregation

geeknarrator opened a new pull request, #8781:
URL: https://github.com/apache/pinot/pull/8781

   Bugfix for: https://github.com/apache/pinot/issues/8226
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] geeknarrator commented on a diff in pull request #8781: Fix Upsert config validation to check for metrics aggregation

Posted by GitBox <gi...@apache.org>.
geeknarrator commented on code in PR #8781:
URL: https://github.com/apache/pinot/pull/8781#discussion_r886350220


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -521,19 +520,32 @@ static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
     Preconditions.checkState(streamConfig.hasLowLevelConsumerType() && !streamConfig.hasHighLevelConsumerType(),
         "Upsert table must use low-level streaming consumer type");
     // replica group is configured for routing
-    Preconditions.checkState(
-        tableConfig.getRoutingConfig() != null && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE
-            .equalsIgnoreCase(tableConfig.getRoutingConfig().getInstanceSelectorType()),
+    Preconditions.checkState(tableConfig.getRoutingConfig() != null
+            && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE.equalsIgnoreCase(
+            tableConfig.getRoutingConfig().getInstanceSelectorType()),
         "Upsert table must use strict replica-group (i.e. strictReplicaGroup) based routing");
     // no startree index
-    Preconditions.checkState(
-        CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs()) && !tableConfig
-            .getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
+    Preconditions.checkState(CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs())
+        && !tableConfig.getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
     // comparison column exists
     if (tableConfig.getUpsertConfig().getComparisonColumn() != null) {
       String comparisonCol = tableConfig.getUpsertConfig().getComparisonColumn();
       Preconditions.checkState(schema.hasColumn(comparisonCol), "The comparison column does not exist on schema");
     }
+    Preconditions.checkState(!isAggregateMetricsEnabled(tableConfig),
+        "Metrics aggregation and upsert cannot be enabled together");
+  }
+
+  /**
+   * Checks if Metrics aggregation is enabled.
+   */
+  @VisibleForTesting
+  static boolean isAggregateMetricsEnabled(TableConfig config) {
+    boolean isAggregateMetricsEnabledInIndexingConfig =
+        Optional.ofNullable(config.getIndexingConfig()).map(IndexingConfig::isAggregateMetrics).orElse(false);
+    boolean hasAggregationConfigs = Optional.ofNullable(config.getIngestionConfig())
+        .map(ingestionConfig -> CollectionUtils.isNotEmpty(ingestionConfig.getAggregationConfigs())).orElse(false);
+    return isAggregateMetricsEnabledInIndexingConfig || hasAggregationConfigs;

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8781: Fix Upsert config validation to check for metrics aggregation

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #8781:
URL: https://github.com/apache/pinot/pull/8781#discussion_r883174305


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -534,6 +534,19 @@ static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
       String comparisonCol = tableConfig.getUpsertConfig().getComparisonColumn();
       Preconditions.checkState(schema.hasColumn(comparisonCol), "The comparison column does not exist on schema");
     }
+    }
+    Preconditions.checkState(isAggregateMetricsEnabled(tableConfig),
+                "Metrics aggregation and upsert cannot be enabled together");
+  }
+
+  /**
+      * Checks if Metrics aggregation is enabled.
+      */
+  @VisibleForTesting
+     static boolean isAggregateMetricsEnabled(TableConfig config) {
+      if(config.getIndexingConfig() == null || config.getIngestionConfig() == null) {return false;
+      }
+      return config.getIndexingConfig().isAggregateMetrics() && CollectionUtils.isEmpty(config.getIngestionConfig().getAggregationConfigs());

Review Comment:
   This should be `||` instead of `&&`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] geeknarrator commented on pull request #8781: Fix Upsert config validation to check for metrics aggregation

Posted by GitBox <gi...@apache.org>.
geeknarrator commented on PR #8781:
URL: https://github.com/apache/pinot/pull/8781#issuecomment-1140296153

   > I don't think it can compile now because of the extra back brace. Can you please also fix the code format using [Pinot Style](https://docs.pinot.apache.org/developers/developers-and-contributors/code-setup#setup-ide)?
   
   Thanks, I configured Pinot style. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] geeknarrator commented on pull request #8781: Fix Upsert config validation to check for metrics aggregation

Posted by GitBox <gi...@apache.org>.
geeknarrator commented on PR #8781:
URL: https://github.com/apache/pinot/pull/8781#issuecomment-1143158820

   @Jackie-Jiang - Thanks for reviewing the changes :) I have addressed the remaining comments. Once you approve those, can I get it merged? I don't seem to have the access. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #8781: Fix Upsert config validation to check for metrics aggregation

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #8781:
URL: https://github.com/apache/pinot/pull/8781#discussion_r886043582


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -521,19 +520,32 @@ static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
     Preconditions.checkState(streamConfig.hasLowLevelConsumerType() && !streamConfig.hasHighLevelConsumerType(),
         "Upsert table must use low-level streaming consumer type");
     // replica group is configured for routing
-    Preconditions.checkState(
-        tableConfig.getRoutingConfig() != null && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE
-            .equalsIgnoreCase(tableConfig.getRoutingConfig().getInstanceSelectorType()),
+    Preconditions.checkState(tableConfig.getRoutingConfig() != null
+            && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE.equalsIgnoreCase(
+            tableConfig.getRoutingConfig().getInstanceSelectorType()),
         "Upsert table must use strict replica-group (i.e. strictReplicaGroup) based routing");
     // no startree index
-    Preconditions.checkState(
-        CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs()) && !tableConfig
-            .getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
+    Preconditions.checkState(CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs())
+        && !tableConfig.getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
     // comparison column exists
     if (tableConfig.getUpsertConfig().getComparisonColumn() != null) {
       String comparisonCol = tableConfig.getUpsertConfig().getComparisonColumn();
       Preconditions.checkState(schema.hasColumn(comparisonCol), "The comparison column does not exist on schema");
     }
+    Preconditions.checkState(!isAggregateMetricsEnabled(tableConfig),
+        "Metrics aggregation and upsert cannot be enabled together");
+  }
+
+  /**
+   * Checks if Metrics aggregation is enabled.
+   */
+  @VisibleForTesting
+  static boolean isAggregateMetricsEnabled(TableConfig config) {
+    boolean isAggregateMetricsEnabledInIndexingConfig =
+        Optional.ofNullable(config.getIndexingConfig()).map(IndexingConfig::isAggregateMetrics).orElse(false);
+    boolean hasAggregationConfigs = Optional.ofNullable(config.getIngestionConfig())
+        .map(ingestionConfig -> CollectionUtils.isNotEmpty(ingestionConfig.getAggregationConfigs())).orElse(false);
+    return isAggregateMetricsEnabledInIndexingConfig || hasAggregationConfigs;

Review Comment:
   Let's validate both configs cannot be enabled at the same time



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -521,19 +520,32 @@ static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
     Preconditions.checkState(streamConfig.hasLowLevelConsumerType() && !streamConfig.hasHighLevelConsumerType(),
         "Upsert table must use low-level streaming consumer type");
     // replica group is configured for routing
-    Preconditions.checkState(
-        tableConfig.getRoutingConfig() != null && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE
-            .equalsIgnoreCase(tableConfig.getRoutingConfig().getInstanceSelectorType()),
+    Preconditions.checkState(tableConfig.getRoutingConfig() != null
+            && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE.equalsIgnoreCase(
+            tableConfig.getRoutingConfig().getInstanceSelectorType()),
         "Upsert table must use strict replica-group (i.e. strictReplicaGroup) based routing");
     // no startree index
-    Preconditions.checkState(
-        CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs()) && !tableConfig
-            .getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
+    Preconditions.checkState(CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs())
+        && !tableConfig.getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
     // comparison column exists
     if (tableConfig.getUpsertConfig().getComparisonColumn() != null) {
       String comparisonCol = tableConfig.getUpsertConfig().getComparisonColumn();
       Preconditions.checkState(schema.hasColumn(comparisonCol), "The comparison column does not exist on schema");
     }
+    Preconditions.checkState(!isAggregateMetricsEnabled(tableConfig),
+        "Metrics aggregation and upsert cannot be enabled together");
+  }
+
+  /**
+   * Checks if Metrics aggregation is enabled.
+   */
+  @VisibleForTesting
+  static boolean isAggregateMetricsEnabled(TableConfig config) {
+    boolean isAggregateMetricsEnabledInIndexingConfig =
+        Optional.ofNullable(config.getIndexingConfig()).map(IndexingConfig::isAggregateMetrics).orElse(false);
+    boolean hasAggregationConfigs = Optional.ofNullable(config.getIngestionConfig())
+        .map(ingestionConfig -> CollectionUtils.isNotEmpty(ingestionConfig.getAggregationConfigs())).orElse(false);

Review Comment:
   Suggest using the non-functional apis which performs better and more readable
   ```suggestion
       boolean isAggregateMetricsEnabledInIndexingConfig = config.getIndexingConfig().isAggregateMetrics();
       boolean hasAggregationConfigs = config.getIngestionConfig() != null && CollectionUtils.isNotEmpty(config.getIngestionConfig().getAggregationConfigs());
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang merged pull request #8781: Fix Upsert config validation to check for metrics aggregation

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged PR #8781:
URL: https://github.com/apache/pinot/pull/8781


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #8781: Fix Upsert config validation to check for metrics aggregation

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #8781:
URL: https://github.com/apache/pinot/pull/8781#issuecomment-1144043889

   # [Codecov](https://codecov.io/gh/apache/pinot/pull/8781?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#8781](https://codecov.io/gh/apache/pinot/pull/8781?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bc5b6d2) into [master](https://codecov.io/gh/apache/pinot/commit/73316a6cf7016bf028b002a93b27da1391ce88dd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (73316a6) will **decrease** coverage by `42.96%`.
   > The diff coverage is `61.23%`.
   
   > :exclamation: Current head bc5b6d2 differs from pull request most recent head 7e7e717. Consider uploading reports for the commit 7e7e717 to get more accurate results
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #8781       +/-   ##
   =============================================
   - Coverage     68.51%   25.54%   -42.97%     
   + Complexity     4601       45     -4556     
   =============================================
     Files          1736     1723       -13     
     Lines         91196    90966      -230     
     Branches      13633    13610       -23     
   =============================================
   - Hits          62480    23240    -39240     
   - Misses        24330    65505    +41175     
   + Partials       4386     2221     -2165     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `25.54% <61.23%> (?)` | |
   | unittests1 | `?` | |
   | unittests2 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8781?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...roker/requesthandler/BaseBrokerRequestHandler.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvQmFzZUJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `61.59% <0.00%> (-5.23%)` | :arrow_down: |
   | [...uting/segmentselector/RealtimeSegmentSelector.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50c2VsZWN0b3IvUmVhbHRpbWVTZWdtZW50U2VsZWN0b3IuamF2YQ==) | `88.13% <0.00%> (-8.69%)` | :arrow_down: |
   | [...he/pinot/common/tier/FixedTierSegmentSelector.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdGllci9GaXhlZFRpZXJTZWdtZW50U2VsZWN0b3IuamF2YQ==) | `0.00% <0.00%> (-93.34%)` | :arrow_down: |
   | [.../helix/core/realtime/SegmentCompletionManager.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1NlZ21lbnRDb21wbGV0aW9uTWFuYWdlci5qYXZh) | `37.47% <0.00%> (-34.54%)` | :arrow_down: |
   | [.../org/apache/pinot/core/common/MinionConstants.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vTWluaW9uQ29uc3RhbnRzLmphdmE=) | `0.00% <ø> (ø)` | |
   | [...t/core/plan/AggregationGroupByOrderByPlanNode.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uR3JvdXBCeU9yZGVyQnlQbGFuTm9kZS5qYXZh) | `45.45% <0.00%> (-36.37%)` | :arrow_down: |
   | [...ot/plugin/stream/kafka20/RowMetadataExtractor.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1zdHJlYW0taW5nZXN0aW9uL3Bpbm90LWthZmthLTIuMC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcGx1Z2luL3N0cmVhbS9rYWZrYTIwL1Jvd01ldGFkYXRhRXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...l/indexsegment/immutable/ImmutableSegmentImpl.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRJbXBsLmphdmE=) | `0.00% <0.00%> (-65.63%)` | :arrow_down: |
   | [...eaderwriter/RealtimeIndexOffHeapMemoryManager.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pby9yZWFkZXJ3cml0ZXIvUmVhbHRpbWVJbmRleE9mZkhlYXBNZW1vcnlNYW5hZ2VyLmphdmE=) | `0.00% <0.00%> (-80.65%)` | :arrow_down: |
   | [...dex/readers/bloom/GuavaBloomFilterReaderUtils.java](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvYmxvb20vR3VhdmFCbG9vbUZpbHRlclJlYWRlclV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1344 more](https://codecov.io/gh/apache/pinot/pull/8781/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8781?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8781?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [73316a6...7e7e717](https://codecov.io/gh/apache/pinot/pull/8781?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] geeknarrator commented on a diff in pull request #8781: Fix Upsert config validation to check for metrics aggregation

Posted by GitBox <gi...@apache.org>.
geeknarrator commented on code in PR #8781:
URL: https://github.com/apache/pinot/pull/8781#discussion_r886342049


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -521,19 +520,32 @@ static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
     Preconditions.checkState(streamConfig.hasLowLevelConsumerType() && !streamConfig.hasHighLevelConsumerType(),
         "Upsert table must use low-level streaming consumer type");
     // replica group is configured for routing
-    Preconditions.checkState(
-        tableConfig.getRoutingConfig() != null && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE
-            .equalsIgnoreCase(tableConfig.getRoutingConfig().getInstanceSelectorType()),
+    Preconditions.checkState(tableConfig.getRoutingConfig() != null
+            && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE.equalsIgnoreCase(
+            tableConfig.getRoutingConfig().getInstanceSelectorType()),
         "Upsert table must use strict replica-group (i.e. strictReplicaGroup) based routing");
     // no startree index
-    Preconditions.checkState(
-        CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs()) && !tableConfig
-            .getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
+    Preconditions.checkState(CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs())
+        && !tableConfig.getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
     // comparison column exists
     if (tableConfig.getUpsertConfig().getComparisonColumn() != null) {
       String comparisonCol = tableConfig.getUpsertConfig().getComparisonColumn();
       Preconditions.checkState(schema.hasColumn(comparisonCol), "The comparison column does not exist on schema");
     }
+    Preconditions.checkState(!isAggregateMetricsEnabled(tableConfig),
+        "Metrics aggregation and upsert cannot be enabled together");
+  }
+
+  /**
+   * Checks if Metrics aggregation is enabled.
+   */
+  @VisibleForTesting
+  static boolean isAggregateMetricsEnabled(TableConfig config) {
+    boolean isAggregateMetricsEnabledInIndexingConfig =
+        Optional.ofNullable(config.getIndexingConfig()).map(IndexingConfig::isAggregateMetrics).orElse(false);
+    boolean hasAggregationConfigs = Optional.ofNullable(config.getIngestionConfig())
+        .map(ingestionConfig -> CollectionUtils.isNotEmpty(ingestionConfig.getAggregationConfigs())).orElse(false);
+    return isAggregateMetricsEnabledInIndexingConfig || hasAggregationConfigs;

Review Comment:
   I see.. I didn't know that is a requirement. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] geeknarrator commented on a diff in pull request #8781: Fix Upsert config validation to check for metrics aggregation

Posted by GitBox <gi...@apache.org>.
geeknarrator commented on code in PR #8781:
URL: https://github.com/apache/pinot/pull/8781#discussion_r884155819


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -534,6 +534,19 @@ static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
       String comparisonCol = tableConfig.getUpsertConfig().getComparisonColumn();
       Preconditions.checkState(schema.hasColumn(comparisonCol), "The comparison column does not exist on schema");
     }
+    }
+    Preconditions.checkState(isAggregateMetricsEnabled(tableConfig),
+                "Metrics aggregation and upsert cannot be enabled together");
+  }
+
+  /**
+      * Checks if Metrics aggregation is enabled.
+      */
+  @VisibleForTesting
+     static boolean isAggregateMetricsEnabled(TableConfig config) {
+      if(config.getIndexingConfig() == null || config.getIngestionConfig() == null) {return false;
+      }
+      return config.getIndexingConfig().isAggregateMetrics() && CollectionUtils.isEmpty(config.getIngestionConfig().getAggregationConfigs());

Review Comment:
   I think I referred `MutableSegmentImpl` and misunderstood. Fixed it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] geeknarrator commented on pull request #8781: Fix Upsert config validation to check for metrics aggregation

Posted by GitBox <gi...@apache.org>.
geeknarrator commented on PR #8781:
URL: https://github.com/apache/pinot/pull/8781#issuecomment-1146176170

   @Jackie-Jiang - Could you please help me merge this? I have resolved the conflict. (also the formatting changes are done by the pinot checkstyle configuration.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] geeknarrator commented on a diff in pull request #8781: Fix Upsert config validation to check for metrics aggregation

Posted by GitBox <gi...@apache.org>.
geeknarrator commented on code in PR #8781:
URL: https://github.com/apache/pinot/pull/8781#discussion_r886341586


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java:
##########
@@ -521,19 +520,32 @@ static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
     Preconditions.checkState(streamConfig.hasLowLevelConsumerType() && !streamConfig.hasHighLevelConsumerType(),
         "Upsert table must use low-level streaming consumer type");
     // replica group is configured for routing
-    Preconditions.checkState(
-        tableConfig.getRoutingConfig() != null && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE
-            .equalsIgnoreCase(tableConfig.getRoutingConfig().getInstanceSelectorType()),
+    Preconditions.checkState(tableConfig.getRoutingConfig() != null
+            && RoutingConfig.STRICT_REPLICA_GROUP_INSTANCE_SELECTOR_TYPE.equalsIgnoreCase(
+            tableConfig.getRoutingConfig().getInstanceSelectorType()),
         "Upsert table must use strict replica-group (i.e. strictReplicaGroup) based routing");
     // no startree index
-    Preconditions.checkState(
-        CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs()) && !tableConfig
-            .getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
+    Preconditions.checkState(CollectionUtils.isEmpty(tableConfig.getIndexingConfig().getStarTreeIndexConfigs())
+        && !tableConfig.getIndexingConfig().isEnableDefaultStarTree(), "The upsert table cannot have star-tree index.");
     // comparison column exists
     if (tableConfig.getUpsertConfig().getComparisonColumn() != null) {
       String comparisonCol = tableConfig.getUpsertConfig().getComparisonColumn();
       Preconditions.checkState(schema.hasColumn(comparisonCol), "The comparison column does not exist on schema");
     }
+    Preconditions.checkState(!isAggregateMetricsEnabled(tableConfig),
+        "Metrics aggregation and upsert cannot be enabled together");
+  }
+
+  /**
+   * Checks if Metrics aggregation is enabled.
+   */
+  @VisibleForTesting
+  static boolean isAggregateMetricsEnabled(TableConfig config) {
+    boolean isAggregateMetricsEnabledInIndexingConfig =
+        Optional.ofNullable(config.getIndexingConfig()).map(IndexingConfig::isAggregateMetrics).orElse(false);
+    boolean hasAggregationConfigs = Optional.ofNullable(config.getIngestionConfig())
+        .map(ingestionConfig -> CollectionUtils.isNotEmpty(ingestionConfig.getAggregationConfigs())).orElse(false);

Review Comment:
   👍 sgtm



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org