You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/10/15 19:36:30 UTC

[GitHub] [incubator-pinot] yupeng9 opened a new pull request #6149: add validation for upsert tables

yupeng9 opened a new pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149


   ## Description
   Part of a series of PRs for #4261
   Check this [doc](https://docs.google.com/document/d/1qljEMndPMxbbKtjlVn9mn2toz7Qrk0TGQsHLfI--7h8/edit#heading=h.lsfmyoyyxtgt) out for the new design
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] yupeng9 commented on a change in pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#discussion_r506032003



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##########
@@ -230,6 +235,39 @@ private static void validateIngestionConfig(@Nullable IngestionConfig ingestionC
     }
   }
 
+  /**
+   * Validates the upsert-related configurations
+   *  - check table type is realtime
+   *  - the primary key exists on the schema
+   *  - replica group is configured for routing type
+   *  - consumer type must be low-level
+   */
+  protected static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
+    if (tableConfig.getUpsertMode() == UpsertConfig.Mode.NONE) {
+      return;
+    }
+    // check table type is realtime
+    if (tableConfig.getTableType() != TableType.REALTIME) {

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] icefury71 commented on pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
icefury71 commented on pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#issuecomment-709577612


   Can you add unit test for this as well please ? Look at TableConfigUtilsTest


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] yupeng9 commented on a change in pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#discussion_r505877388



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##########
@@ -230,6 +234,38 @@ private static void validateIngestionConfig(@Nullable IngestionConfig ingestionC
     }
   }
 
+  /**
+   * Validates the upsert-related configurations
+   *
+   */
+  private static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
+    if (tableConfig.getUpsertMode() == UpsertConfig.Mode.NONE) {
+      return;
+    }
+    // primary key exists
+    if (schema.getPrimaryKeyColumns() == null || schema.getPrimaryKeyColumns().isEmpty()) {
+      throw new IllegalStateException("Upsert table must have primary key columns in the schema.");
+    }

Review comment:
       I saw some checks in this class use precondition, others use `IllegalStateException`. Shall we make it consistent? @icefury71 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#discussion_r505924807



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##########
@@ -230,6 +235,39 @@ private static void validateIngestionConfig(@Nullable IngestionConfig ingestionC
     }
   }
 
+  /**
+   * Validates the upsert-related configurations
+   *  - check table type is realtime
+   *  - the primary key exists on the schema
+   *  - replica group is configured for routing type
+   *  - consumer type must be low-level
+   */
+  protected static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
+    if (tableConfig.getUpsertMode() == UpsertConfig.Mode.NONE) {
+      return;
+    }
+    // check table type is realtime
+    if (tableConfig.getTableType() != TableType.REALTIME) {

Review comment:
       (nit) Use precondition for this as well




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] yupeng9 commented on a change in pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#discussion_r505884895



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##########
@@ -230,6 +234,38 @@ private static void validateIngestionConfig(@Nullable IngestionConfig ingestionC
     }
   }
 
+  /**
+   * Validates the upsert-related configurations
+   *
+   */
+  private static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
+    if (tableConfig.getUpsertMode() == UpsertConfig.Mode.NONE) {
+      return;
+    }
+    // primary key exists
+    if (schema.getPrimaryKeyColumns() == null || schema.getPrimaryKeyColumns().isEmpty()) {
+      throw new IllegalStateException("Upsert table must have primary key columns in the schema.");
+    }
+    // replica group is configured for routing
+    if (tableConfig.getRoutingConfig() == null || !tableConfig.getRoutingConfig().getInstanceSelectorType()
+        .equalsIgnoreCase(RoutingConfig.REPLICA_GROUP_INSTANCE_SELECTOR_TYPE)) {
+      throw new IllegalStateException("Upsert table must use replicaGroup as the routing config.");
+    }

Review comment:
       why? this seems the same rewrite.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] icefury71 commented on a change in pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
icefury71 commented on a change in pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#discussion_r505828804



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##########
@@ -230,6 +234,38 @@ private static void validateIngestionConfig(@Nullable IngestionConfig ingestionC
     }
   }
 
+  /**
+   * Validates the upsert-related configurations
+   *
+   */
+  private static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
+    if (tableConfig.getUpsertMode() == UpsertConfig.Mode.NONE) {
+      return;
+    }
+    // primary key exists
+    if (schema.getPrimaryKeyColumns() == null || schema.getPrimaryKeyColumns().isEmpty()) {
+      throw new IllegalStateException("Upsert table must have primary key columns in the schema.");
+    }
+    // replica group is configured for routing
+    if (tableConfig.getRoutingConfig() == null || !tableConfig.getRoutingConfig().getInstanceSelectorType()
+        .equalsIgnoreCase(RoutingConfig.REPLICA_GROUP_INSTANCE_SELECTOR_TYPE)) {
+      throw new IllegalStateException("Upsert table must use replicaGroup as the routing config.");
+    }
+    // consumer type must be low-level
+    if (tableConfig.getIndexingConfig() != null && tableConfig.getIndexingConfig().getStreamConfigs() != null) {
+      StreamConfig streamConfig =
+          new StreamConfig(tableConfig.getTableName(), tableConfig.getIndexingConfig().getStreamConfigs());
+      if (streamConfig.getConsumerTypes().size() != 1
+          || streamConfig.getConsumerTypes().get(0) != StreamConfig.ConsumerType.LOWLEVEL) {
+        throw new IllegalStateException("Upsert table must use low-level streaming consumer type.");
+      }
+    }
+    // check type type is realtime
+    if (tableConfig.getTableType() != TableType.REALTIME) {

Review comment:
       nit: do this in the beginning ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#issuecomment-709566750


   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=h1) Report
   > Merging [#6149](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) will **increase** coverage by `6.36%`.
   > The diff coverage is `60.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/6149/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz)](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #6149      +/-   ##
   ==========================================
   + Coverage   66.44%   72.81%   +6.36%     
   ==========================================
     Files        1075     1231     +156     
     Lines       54773    58069    +3296     
     Branches     8168     8573     +405     
   ==========================================
   + Hits        36396    42281    +5885     
   + Misses      15700    13004    -2696     
   - Partials     2677     2784     +107     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | #integration | `45.31% <49.14%> (?)` | |
   | #unittests | `63.91% <38.09%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `100.00% <ø> (ø)` | |
   | [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `52.83% <0.00%> (-13.84%)` | :arrow_down: |
   | [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `53.33% <0.00%> (-3.81%)` | :arrow_down: |
   | [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `44.44% <0.00%> (-4.40%)` | :arrow_down: |
   | [.../org/apache/pinot/client/ResultTableResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L1Jlc3VsdFRhYmxlUmVzdWx0U2V0LmphdmE=) | `24.00% <0.00%> (-10.29%)` | :arrow_down: |
   | [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `78.57% <ø> (+5.40%)` | :arrow_up: |
   | [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
   | [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `98.27% <ø> (-1.73%)` | :arrow_down: |
   | [.../pinot/common/function/DateTimePatternHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRGF0ZVRpbWVQYXR0ZXJuSGFuZGxlci5qYXZh) | `83.33% <ø> (ø)` | |
   | [...ot/common/function/FunctionDefinitionRegistry.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25EZWZpbml0aW9uUmVnaXN0cnkuamF2YQ==) | `88.88% <ø> (+44.44%)` | :arrow_up: |
   | ... and [981 more](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=footer). Last update [a910f5d...d499f7c](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#issuecomment-709566750


   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=h1) Report
   > Merging [#6149](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) will **increase** coverage by `6.29%`.
   > The diff coverage is `60.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/6149/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz)](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #6149      +/-   ##
   ==========================================
   + Coverage   66.44%   72.74%   +6.29%     
   ==========================================
     Files        1075     1231     +156     
     Lines       54773    58072    +3299     
     Branches     8168     8572     +404     
   ==========================================
   + Hits        36396    42245    +5849     
   + Misses      15700    13037    -2663     
   - Partials     2677     2790     +113     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | #integration | `45.09% <49.14%> (?)` | |
   | #unittests | `63.93% <38.09%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `100.00% <ø> (ø)` | |
   | [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `52.83% <0.00%> (-13.84%)` | :arrow_down: |
   | [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `53.33% <0.00%> (-3.81%)` | :arrow_down: |
   | [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `44.44% <0.00%> (-4.40%)` | :arrow_down: |
   | [.../org/apache/pinot/client/ResultTableResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L1Jlc3VsdFRhYmxlUmVzdWx0U2V0LmphdmE=) | `24.00% <0.00%> (-10.29%)` | :arrow_down: |
   | [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `78.57% <ø> (+5.40%)` | :arrow_up: |
   | [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
   | [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `98.27% <ø> (-1.73%)` | :arrow_down: |
   | [.../pinot/common/function/DateTimePatternHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRGF0ZVRpbWVQYXR0ZXJuSGFuZGxlci5qYXZh) | `83.33% <ø> (ø)` | |
   | [...ot/common/function/FunctionDefinitionRegistry.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25EZWZpbml0aW9uUmVnaXN0cnkuamF2YQ==) | `88.88% <ø> (+44.44%)` | :arrow_up: |
   | ... and [982 more](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=footer). Last update [a910f5d...8704e55](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#issuecomment-709566750


   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=h1) Report
   > Merging [#6149](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) will **decrease** coverage by `2.51%`.
   > The diff coverage is `38.09%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/6149/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz)](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #6149      +/-   ##
   ==========================================
   - Coverage   66.44%   63.93%   -2.52%     
   ==========================================
     Files        1075     1231     +156     
     Lines       54773    58072    +3299     
     Branches     8168     8572     +404     
   ==========================================
   + Hits        36396    37129     +733     
   - Misses      15700    18241    +2541     
   - Partials     2677     2702      +25     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | #unittests | `63.93% <38.09%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...e/pinot/broker/api/resources/PinotBrokerDebug.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckRlYnVnLmphdmE=) | `0.00% <0.00%> (-79.32%)` | :arrow_down: |
   | [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `71.42% <ø> (-28.58%)` | :arrow_down: |
   | [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `33.96% <0.00%> (-32.71%)` | :arrow_down: |
   | [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `53.33% <0.00%> (-3.81%)` | :arrow_down: |
   | [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `35.55% <0.00%> (-13.29%)` | :arrow_down: |
   | [...inot/client/JsonAsyncHttpPinotClientTransport.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0pzb25Bc3luY0h0dHBQaW5vdENsaWVudFRyYW5zcG9ydC5qYXZh) | `10.90% <0.00%> (-51.10%)` | :arrow_down: |
   | [.../org/apache/pinot/client/ResultTableResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L1Jlc3VsdFRhYmxlUmVzdWx0U2V0LmphdmE=) | `0.00% <0.00%> (-34.29%)` | :arrow_down: |
   | [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `73.80% <ø> (+0.63%)` | :arrow_up: |
   | [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
   | [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `98.27% <ø> (-1.73%)` | :arrow_down: |
   | ... and [1056 more](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=footer). Last update [a910f5d...8704e55](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#discussion_r505924483



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##########
@@ -230,6 +234,38 @@ private static void validateIngestionConfig(@Nullable IngestionConfig ingestionC
     }
   }
 
+  /**
+   * Validates the upsert-related configurations
+   *
+   */
+  private static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
+    if (tableConfig.getUpsertMode() == UpsertConfig.Mode.NONE) {
+      return;
+    }
+    // primary key exists
+    if (schema.getPrimaryKeyColumns() == null || schema.getPrimaryKeyColumns().isEmpty()) {
+      throw new IllegalStateException("Upsert table must have primary key columns in the schema.");
+    }
+    // replica group is configured for routing
+    if (tableConfig.getRoutingConfig() == null || !tableConfig.getRoutingConfig().getInstanceSelectorType()
+        .equalsIgnoreCase(RoutingConfig.REPLICA_GROUP_INSTANCE_SELECTOR_TYPE)) {
+      throw new IllegalStateException("Upsert table must use replicaGroup as the routing config.");
+    }

Review comment:
       `tableConfig.getRoutingConfig().getInstanceSelectorType()` might be `null`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] yupeng9 commented on a change in pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#discussion_r505885977



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##########
@@ -230,6 +234,38 @@ private static void validateIngestionConfig(@Nullable IngestionConfig ingestionC
     }
   }
 
+  /**
+   * Validates the upsert-related configurations
+   *
+   */
+  private static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
+    if (tableConfig.getUpsertMode() == UpsertConfig.Mode.NONE) {
+      return;
+    }
+    // primary key exists
+    if (schema.getPrimaryKeyColumns() == null || schema.getPrimaryKeyColumns().isEmpty()) {
+      throw new IllegalStateException("Upsert table must have primary key columns in the schema.");
+    }
+    // replica group is configured for routing
+    if (tableConfig.getRoutingConfig() == null || !tableConfig.getRoutingConfig().getInstanceSelectorType()
+        .equalsIgnoreCase(RoutingConfig.REPLICA_GROUP_INSTANCE_SELECTOR_TYPE)) {
+      throw new IllegalStateException("Upsert table must use replicaGroup as the routing config.");
+    }
+    // consumer type must be low-level
+    if (tableConfig.getIndexingConfig() != null && tableConfig.getIndexingConfig().getStreamConfigs() != null) {
+      StreamConfig streamConfig =
+          new StreamConfig(tableConfig.getTableName(), tableConfig.getIndexingConfig().getStreamConfigs());
+      if (streamConfig.getConsumerTypes().size() != 1
+          || streamConfig.getConsumerTypes().get(0) != StreamConfig.ConsumerType.LOWLEVEL) {
+        throw new IllegalStateException("Upsert table must use low-level streaming consumer type.");
+      }
+    }
+    // check type type is realtime
+    if (tableConfig.getTableType() != TableType.REALTIME) {

Review comment:
       sgtm




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-io commented on pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#issuecomment-709566750


   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=h1) Report
   > Merging [#6149](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) will **decrease** coverage by `2.53%`.
   > The diff coverage is `38.09%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/6149/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz)](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #6149      +/-   ##
   ==========================================
   - Coverage   66.44%   63.91%   -2.54%     
   ==========================================
     Files        1075     1231     +156     
     Lines       54773    58069    +3296     
     Branches     8168     8573     +405     
   ==========================================
   + Hits        36396    37114     +718     
   - Misses      15700    18253    +2553     
   - Partials     2677     2702      +25     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | #unittests | `63.91% <38.09%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...e/pinot/broker/api/resources/PinotBrokerDebug.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckRlYnVnLmphdmE=) | `0.00% <0.00%> (-79.32%)` | :arrow_down: |
   | [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `71.42% <ø> (-28.58%)` | :arrow_down: |
   | [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `33.96% <0.00%> (-32.71%)` | :arrow_down: |
   | [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `53.33% <0.00%> (-3.81%)` | :arrow_down: |
   | [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `35.55% <0.00%> (-13.29%)` | :arrow_down: |
   | [...inot/client/JsonAsyncHttpPinotClientTransport.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0pzb25Bc3luY0h0dHBQaW5vdENsaWVudFRyYW5zcG9ydC5qYXZh) | `10.90% <0.00%> (-51.10%)` | :arrow_down: |
   | [.../org/apache/pinot/client/ResultTableResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L1Jlc3VsdFRhYmxlUmVzdWx0U2V0LmphdmE=) | `0.00% <0.00%> (-34.29%)` | :arrow_down: |
   | [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `73.80% <ø> (+0.63%)` | :arrow_up: |
   | [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
   | [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `98.27% <ø> (-1.73%)` | :arrow_down: |
   | ... and [1057 more](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=footer). Last update [a910f5d...d499f7c](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] yupeng9 commented on pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#issuecomment-709626207


   @icefury71 @Jackie-Jiang thanks for the review. Comments addressed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#issuecomment-709566750


   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=h1) Report
   > Merging [#6149](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) will **increase** coverage by `6.34%`.
   > The diff coverage is `60.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/6149/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz)](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #6149      +/-   ##
   ==========================================
   + Coverage   66.44%   72.79%   +6.34%     
   ==========================================
     Files        1075     1231     +156     
     Lines       54773    58067    +3294     
     Branches     8168     8571     +403     
   ==========================================
   + Hits        36396    42268    +5872     
   + Misses      15700    13011    -2689     
   - Partials     2677     2788     +111     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | #integration | `45.21% <49.14%> (?)` | |
   | #unittests | `63.95% <38.09%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `100.00% <ø> (ø)` | |
   | [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `52.83% <0.00%> (-13.84%)` | :arrow_down: |
   | [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `53.33% <0.00%> (-3.81%)` | :arrow_down: |
   | [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `44.44% <0.00%> (-4.40%)` | :arrow_down: |
   | [.../org/apache/pinot/client/ResultTableResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L1Jlc3VsdFRhYmxlUmVzdWx0U2V0LmphdmE=) | `24.00% <0.00%> (-10.29%)` | :arrow_down: |
   | [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `78.57% <ø> (+5.40%)` | :arrow_up: |
   | [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
   | [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `98.27% <ø> (-1.73%)` | :arrow_down: |
   | [.../pinot/common/function/DateTimePatternHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRGF0ZVRpbWVQYXR0ZXJuSGFuZGxlci5qYXZh) | `83.33% <ø> (ø)` | |
   | [...ot/common/function/FunctionDefinitionRegistry.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25EZWZpbml0aW9uUmVnaXN0cnkuamF2YQ==) | `88.88% <ø> (+44.44%)` | :arrow_up: |
   | ... and [982 more](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=footer). Last update [a910f5d...12e2ab5](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#issuecomment-709566750


   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=h1) Report
   > Merging [#6149](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) will **decrease** coverage by `21.23%`.
   > The diff coverage is `49.14%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/6149/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz)](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master    #6149       +/-   ##
   ===========================================
   - Coverage   66.44%   45.21%   -21.24%     
   ===========================================
     Files        1075     1231      +156     
     Lines       54773    58067     +3294     
     Branches     8168     8571      +403     
   ===========================================
   - Hits        36396    26257    -10139     
   - Misses      15700    29649    +13949     
   + Partials     2677     2161      -516     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | #integration | `45.21% <49.14%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `100.00% <ø> (ø)` | |
   | [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `52.83% <0.00%> (-13.84%)` | :arrow_down: |
   | [...org/apache/pinot/broker/queryquota/HitCounter.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9IaXRDb3VudGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...che/pinot/broker/queryquota/MaxHitRateTracker.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9NYXhIaXRSYXRlVHJhY2tlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...ache/pinot/broker/queryquota/QueryQuotaEntity.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9RdWVyeVF1b3RhRW50aXR5LmphdmE=) | `0.00% <0.00%> (-50.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `26.66% <0.00%> (-30.48%)` | :arrow_down: |
   | [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `22.22% <0.00%> (-26.62%)` | :arrow_down: |
   | [.../org/apache/pinot/client/ResultTableResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L1Jlc3VsdFRhYmxlUmVzdWx0U2V0LmphdmE=) | `24.00% <0.00%> (-10.29%)` | :arrow_down: |
   | [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `64.28% <ø> (-8.89%)` | :arrow_down: |
   | [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
   | ... and [1210 more](https://codecov.io/gh/apache/incubator-pinot/pull/6149/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=footer). Last update [a910f5d...12e2ab5](https://codecov.io/gh/apache/incubator-pinot/pull/6149?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] yupeng9 commented on a change in pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#discussion_r506031794



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##########
@@ -230,6 +234,38 @@ private static void validateIngestionConfig(@Nullable IngestionConfig ingestionC
     }
   }
 
+  /**
+   * Validates the upsert-related configurations
+   *
+   */
+  private static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
+    if (tableConfig.getUpsertMode() == UpsertConfig.Mode.NONE) {
+      return;
+    }
+    // primary key exists
+    if (schema.getPrimaryKeyColumns() == null || schema.getPrimaryKeyColumns().isEmpty()) {
+      throw new IllegalStateException("Upsert table must have primary key columns in the schema.");
+    }
+    // replica group is configured for routing
+    if (tableConfig.getRoutingConfig() == null || !tableConfig.getRoutingConfig().getInstanceSelectorType()
+        .equalsIgnoreCase(RoutingConfig.REPLICA_GROUP_INSTANCE_SELECTOR_TYPE)) {
+      throw new IllegalStateException("Upsert table must use replicaGroup as the routing config.");
+    }

Review comment:
       then it's the same :)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] icefury71 commented on a change in pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
icefury71 commented on a change in pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#discussion_r505909024



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##########
@@ -230,6 +234,38 @@ private static void validateIngestionConfig(@Nullable IngestionConfig ingestionC
     }
   }
 
+  /**
+   * Validates the upsert-related configurations
+   *
+   */
+  private static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
+    if (tableConfig.getUpsertMode() == UpsertConfig.Mode.NONE) {
+      return;
+    }
+    // primary key exists
+    if (schema.getPrimaryKeyColumns() == null || schema.getPrimaryKeyColumns().isEmpty()) {
+      throw new IllegalStateException("Upsert table must have primary key columns in the schema.");
+    }

Review comment:
       Will do so in my next pass




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6149: add validation for upsert tables

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6149:
URL: https://github.com/apache/incubator-pinot/pull/6149#discussion_r505814806



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##########
@@ -230,6 +234,38 @@ private static void validateIngestionConfig(@Nullable IngestionConfig ingestionC
     }
   }
 
+  /**
+   * Validates the upsert-related configurations
+   *
+   */
+  private static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
+    if (tableConfig.getUpsertMode() == UpsertConfig.Mode.NONE) {
+      return;
+    }
+    // primary key exists
+    if (schema.getPrimaryKeyColumns() == null || schema.getPrimaryKeyColumns().isEmpty()) {
+      throw new IllegalStateException("Upsert table must have primary key columns in the schema.");
+    }

Review comment:
       ```suggestion
       Preconditions.checkState(CollectionUtils.isNotEmpty(schema.getPrimaryKeyColumns()), "Upsert table must have primary key columns in the schema");
   ```

##########
File path: pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##########
@@ -230,6 +234,38 @@ private static void validateIngestionConfig(@Nullable IngestionConfig ingestionC
     }
   }
 
+  /**
+   * Validates the upsert-related configurations
+   *
+   */
+  private static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
+    if (tableConfig.getUpsertMode() == UpsertConfig.Mode.NONE) {
+      return;
+    }
+    // primary key exists
+    if (schema.getPrimaryKeyColumns() == null || schema.getPrimaryKeyColumns().isEmpty()) {
+      throw new IllegalStateException("Upsert table must have primary key columns in the schema.");
+    }
+    // replica group is configured for routing
+    if (tableConfig.getRoutingConfig() == null || !tableConfig.getRoutingConfig().getInstanceSelectorType()
+        .equalsIgnoreCase(RoutingConfig.REPLICA_GROUP_INSTANCE_SELECTOR_TYPE)) {
+      throw new IllegalStateException("Upsert table must use replicaGroup as the routing config.");
+    }
+    // consumer type must be low-level
+    if (tableConfig.getIndexingConfig() != null && tableConfig.getIndexingConfig().getStreamConfigs() != null) {
+      StreamConfig streamConfig =
+          new StreamConfig(tableConfig.getTableName(), tableConfig.getIndexingConfig().getStreamConfigs());
+      if (streamConfig.getConsumerTypes().size() != 1
+          || streamConfig.getConsumerTypes().get(0) != StreamConfig.ConsumerType.LOWLEVEL) {
+        throw new IllegalStateException("Upsert table must use low-level streaming consumer type.");
+      }

Review comment:
       ```suggestion
         Preconditions.checkState(streamConfig.hasLowLevelConsumerType() && !streamConfig.hasHighLevelConsumerType(), ""Upsert table must use low-level streaming consumer type");
   ```

##########
File path: pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##########
@@ -230,6 +234,38 @@ private static void validateIngestionConfig(@Nullable IngestionConfig ingestionC
     }
   }
 
+  /**
+   * Validates the upsert-related configurations
+   *

Review comment:
       Please add some javadoc on the checks performed here

##########
File path: pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##########
@@ -230,6 +234,38 @@ private static void validateIngestionConfig(@Nullable IngestionConfig ingestionC
     }
   }
 
+  /**
+   * Validates the upsert-related configurations
+   *
+   */
+  private static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
+    if (tableConfig.getUpsertMode() == UpsertConfig.Mode.NONE) {
+      return;
+    }
+    // primary key exists
+    if (schema.getPrimaryKeyColumns() == null || schema.getPrimaryKeyColumns().isEmpty()) {
+      throw new IllegalStateException("Upsert table must have primary key columns in the schema.");
+    }
+    // replica group is configured for routing
+    if (tableConfig.getRoutingConfig() == null || !tableConfig.getRoutingConfig().getInstanceSelectorType()
+        .equalsIgnoreCase(RoutingConfig.REPLICA_GROUP_INSTANCE_SELECTOR_TYPE)) {
+      throw new IllegalStateException("Upsert table must use replicaGroup as the routing config.");
+    }

Review comment:
       This can potentially have NPE
   ```suggestion
       Preconditions.checkState(tableConfig.getRoutingConfig() != null && RoutingConfig.REPLICA_GROUP_INSTANCE_SELECTOR_TYPE. equalsIgnoreCase(tableConfig.getRoutingConfig().getInstanceSelectorType()), "Upsert table must use replica-group based routing")
   ```

##########
File path: pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##########
@@ -230,6 +234,38 @@ private static void validateIngestionConfig(@Nullable IngestionConfig ingestionC
     }
   }
 
+  /**
+   * Validates the upsert-related configurations
+   *
+   */
+  private static void validateUpsertConfig(TableConfig tableConfig, Schema schema) {
+    if (tableConfig.getUpsertMode() == UpsertConfig.Mode.NONE) {
+      return;
+    }
+    // primary key exists
+    if (schema.getPrimaryKeyColumns() == null || schema.getPrimaryKeyColumns().isEmpty()) {
+      throw new IllegalStateException("Upsert table must have primary key columns in the schema.");
+    }
+    // replica group is configured for routing
+    if (tableConfig.getRoutingConfig() == null || !tableConfig.getRoutingConfig().getInstanceSelectorType()
+        .equalsIgnoreCase(RoutingConfig.REPLICA_GROUP_INSTANCE_SELECTOR_TYPE)) {
+      throw new IllegalStateException("Upsert table must use replicaGroup as the routing config.");
+    }
+    // consumer type must be low-level
+    if (tableConfig.getIndexingConfig() != null && tableConfig.getIndexingConfig().getStreamConfigs() != null) {
+      StreamConfig streamConfig =
+          new StreamConfig(tableConfig.getTableName(), tableConfig.getIndexingConfig().getStreamConfigs());
+      if (streamConfig.getConsumerTypes().size() != 1
+          || streamConfig.getConsumerTypes().get(0) != StreamConfig.ConsumerType.LOWLEVEL) {
+        throw new IllegalStateException("Upsert table must use low-level streaming consumer type.");
+      }
+    }
+    // check type type is realtime
+    if (tableConfig.getTableType() != TableType.REALTIME) {

Review comment:
       Use precondition, also recommend reorder the checks in this order: upsert mode -> primary key -> table type -> consumer type -> routing type, which IMO is more intuitive




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org