You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/10/02 20:24:36 UTC

[GitHub] [incubator-pinot] yupeng9 opened a new pull request #6096: add upsert related configs

yupeng9 opened a new pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096


   
   ## Description
   Part of a series of PRs for https://github.com/apache/incubator-pinot/issues/4261
   Check this [doc](https://docs.google.com/document/d/1qljEMndPMxbbKtjlVn9mn2toz7Qrk0TGQsHLfI--7h8/edit#heading=h.lsfmyoyyxtgt) out for the new design
   
    - Added the primary key definition in the schema
    - Changed the upsert config in table config. For the first phase, only support the full update mode.
   
   Does this PR otherwise need attention when creating release notes? Things to consider:
   - New configuration options
   configurations related to upsert
   
   
   
   If you have a series of commits adding or enabling a feature, then
   add this section only in final commit that marks the feature completed.
   Refer to earlier release notes to see examples of text
   
   ## Documentation
   If you have introduced a new feature or configuration, please add it to the documentation as well.
   See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-commenter commented on pull request #6096: add upsert related configs

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#issuecomment-703198725


   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=h1) Report
   > Merging [#6096](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) will **decrease** coverage by `20.98%`.
   > The diff coverage is `48.65%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/6096/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz)](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master    #6096       +/-   ##
   ===========================================
   - Coverage   66.44%   45.45%   -20.99%     
   ===========================================
     Files        1075     1223      +148     
     Lines       54773    57731     +2958     
     Branches     8168     8518      +350     
   ===========================================
   - Hits        36396    26244    -10152     
   - Misses      15700    29351    +13651     
   + Partials     2677     2136      -541     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | #integration | `45.45% <48.65%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `100.00% <ø> (ø)` | |
   | [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `52.83% <0.00%> (-13.84%)` | :arrow_down: |
   | [...org/apache/pinot/broker/queryquota/HitCounter.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9IaXRDb3VudGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...che/pinot/broker/queryquota/MaxHitRateTracker.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9NYXhIaXRSYXRlVHJhY2tlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...ache/pinot/broker/queryquota/QueryQuotaEntity.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9RdWVyeVF1b3RhRW50aXR5LmphdmE=) | `0.00% <0.00%> (-50.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `26.66% <0.00%> (-30.48%)` | :arrow_down: |
   | [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `22.22% <0.00%> (-26.62%)` | :arrow_down: |
   | [.../org/apache/pinot/client/ResultTableResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L1Jlc3VsdFRhYmxlUmVzdWx0U2V0LmphdmE=) | `24.00% <0.00%> (-10.29%)` | :arrow_down: |
   | [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `64.28% <ø> (-8.89%)` | :arrow_down: |
   | [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
   | ... and [1189 more](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=footer). Last update [9929dad...d47729b](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] yupeng9 commented on pull request #6096: add upsert related configs

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#issuecomment-703188624


   I don't think so. Strictly speaking, primary key is a constraint defined on the schema, just like other constraints over the schema such as a column is `NOT NULL` or a column is `UNIQUE`. This is well defined in the database literature.
   
   If the consideration is to limit its usage, then we can add a checker to ensure it's defined only when Upsert mode is not `NONE`. 
   However, for cleaniness (and also the ease of future development and structuring), the constraints shall be defined as part of the schema.
   
   > After some consideration, I still think it is better to wrap the primary key columns inside the `UpsertConfig` for the following reasons:
   > 
   > * The schema for the offline segment is re-constructed from the segment metadata, where primary key columns might not be maintained
   > * For regular use cases, there is no primary key concept in Pinot. Putting it in `UpsertConfig` can keep all related configs at the same place
   > * For the upcoming JOIN feature, we should have another config for the in-memory lookup table, where we can put the primary key there. This 2 features cannot be applied to the same table, and I don't see the benefit of keeping this info inside the schema
   > 
   > Since this is the first feature using the primary key concept, I suggest putting all the related configs at the same place for easier management. If we find this as a common concept in the future, we can move it then.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] yupeng9 commented on a change in pull request #6096: add upsert related configs

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#discussion_r499107805



##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/UpsertConfig.java
##########
@@ -27,44 +27,22 @@
 
 
 public class UpsertConfig extends BaseJsonConfig {
-  // names of the columns that used as primary keys of an upsert table
-  private final List<String> _primaryKeyColumns;
-  // name of the column that we are going to store the offset value to
-  private final String _offsetColumn;
-  // name of the virtual column that we are going to store the validFrom value to
-  private final String _validFromColumn;
-  // name of the virtual column that we are going to store the validUntil value to
-  private final String _validUntilColumn;
+  public static final String MODE_KEY = "mode";
 
-  @JsonCreator
-  public UpsertConfig(@JsonProperty(value = "primaryKeyColumns") List<String> primaryKeyColumns,
-      @JsonProperty(value = "offsetColumn") String offsetColumn,
-      @JsonProperty(value = "validFromColumn") String validFromColumn,
-      @JsonProperty(value = "validUntilColumn") String validUntilColumn) {
-    Preconditions.checkArgument(primaryKeyColumns != null && primaryKeyColumns.size() == 1,
-        "'primaryKeyColumns' must be configured with exact one column");
-    Preconditions.checkArgument(StringUtils.isNotEmpty(offsetColumn), "'offsetColumn' must be configured");
-    Preconditions.checkArgument(StringUtils.isNotEmpty(validFromColumn), "'validFromColumn' must be configured");
-    Preconditions.checkArgument(StringUtils.isNotEmpty(validUntilColumn), "'validUntilColumn' must be configured");
-    _primaryKeyColumns = primaryKeyColumns;
-    _offsetColumn = offsetColumn;
-    _validFromColumn = validFromColumn;
-    _validUntilColumn = validUntilColumn;
+  public enum Mode {
+    FULL, PARTIAL, NONE

Review comment:
       In the implementation, it's more clearer (and easier) to have a value to represent that upsert is disabled. Also, when UpsertConfig is null, it implies the mode is `NONE`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6096: add upsert related configs

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#discussion_r499105212



##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/UpsertConfig.java
##########
@@ -27,44 +27,22 @@
 
 
 public class UpsertConfig extends BaseJsonConfig {
-  // names of the columns that used as primary keys of an upsert table
-  private final List<String> _primaryKeyColumns;
-  // name of the column that we are going to store the offset value to
-  private final String _offsetColumn;
-  // name of the virtual column that we are going to store the validFrom value to
-  private final String _validFromColumn;
-  // name of the virtual column that we are going to store the validUntil value to
-  private final String _validUntilColumn;
+  public static final String MODE_KEY = "mode";
 
-  @JsonCreator
-  public UpsertConfig(@JsonProperty(value = "primaryKeyColumns") List<String> primaryKeyColumns,
-      @JsonProperty(value = "offsetColumn") String offsetColumn,
-      @JsonProperty(value = "validFromColumn") String validFromColumn,
-      @JsonProperty(value = "validUntilColumn") String validUntilColumn) {
-    Preconditions.checkArgument(primaryKeyColumns != null && primaryKeyColumns.size() == 1,
-        "'primaryKeyColumns' must be configured with exact one column");
-    Preconditions.checkArgument(StringUtils.isNotEmpty(offsetColumn), "'offsetColumn' must be configured");
-    Preconditions.checkArgument(StringUtils.isNotEmpty(validFromColumn), "'validFromColumn' must be configured");
-    Preconditions.checkArgument(StringUtils.isNotEmpty(validUntilColumn), "'validUntilColumn' must be configured");
-    _primaryKeyColumns = primaryKeyColumns;
-    _offsetColumn = offsetColumn;
-    _validFromColumn = validFromColumn;
-    _validUntilColumn = validUntilColumn;
+  public enum Mode {
+    FULL, PARTIAL, NONE

Review comment:
       Recommend not putting `NONE` as a mode. Upsert is disabled when the `UpsertConfig` is `null` in the table config (more intuitive IMO)

##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/data/Schema.java
##########
@@ -66,6 +66,9 @@
   private TimeFieldSpec _timeFieldSpec;
   private final List<DateTimeFieldSpec> _dateTimeFieldSpecs = new ArrayList<>();
   private final List<ComplexFieldSpec> _complexFieldSpecs = new ArrayList<>();
+  // names of the columns that used as primary keys
+  // TODO: add validation checks like duplicate columns and use of time column
+  private List<String> _primaryKeyColumns;

Review comment:
       This should not be part of the schema, but the upsert config

##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/TableConfig.java
##########
@@ -265,4 +266,9 @@ public void setIngestionConfig(IngestionConfig ingestionConfig) {
   public void setTierConfigsList(List<TierConfig> tierConfigsList) {
     _tierConfigsList = tierConfigsList;
   }
+
+  @JsonIgnore
+  public UpsertConfig.Mode getUpsertMode() {

Review comment:
       Not sure if this method is needed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] yupeng9 commented on a change in pull request #6096: add upsert related configs

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#discussion_r499107725



##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/data/Schema.java
##########
@@ -66,6 +66,9 @@
   private TimeFieldSpec _timeFieldSpec;
   private final List<DateTimeFieldSpec> _dateTimeFieldSpecs = new ArrayList<>();
   private final List<ComplexFieldSpec> _complexFieldSpecs = new ArrayList<>();
+  // names of the columns that used as primary keys
+  // TODO: add validation checks like duplicate columns and use of time column
+  private List<String> _primaryKeyColumns;

Review comment:
       Primary key is an inherent attribute of the schema. And its usage is beyond the upsert. For example, it's also used in the join feature which is happening in parallel.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] yupeng9 edited a comment on pull request #6096: add upsert related configs

Posted by GitBox <gi...@apache.org>.
yupeng9 edited a comment on pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#issuecomment-703194819


   > After some consideration, I still think it is better to wrap the primary key columns inside the `UpsertConfig` for the following reasons:
   > 
   > * The schema for the offline segment is re-constructed from the segment metadata, where primary key columns might not be maintained
   
   This is fine for now. As upsert applies to real-time segment only. In the future, I think we shall consider adding the primary ey concept to offline segment too
   
   > * For regular use cases, there is no primary key concept in Pinot. Putting it in `UpsertConfig` can keep all related configs at the same place
   > * For the upcoming JOIN feature, we should have another config for the in-memory lookup table, where we can put the primary key there. This 2 features cannot be applied to the same table, and I don't see the benefit of keeping this info inside the schema
   
   For fact-dim join, the primary key actually needs to be defined on the regular table.
   
   > 
   > Since this is the first feature using the primary key concept, I suggest putting all the related configs at the same place for easier management. If we find this as a common concept in the future, we can move it then.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] yupeng9 commented on pull request #6096: add upsert related configs

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#issuecomment-703195721


   @Jackie-Jiang thanks for taking a look. comments addressed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #6096: add upsert related configs

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6096: add upsert related configs

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#issuecomment-703198725


   # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=h1) Report
   > Merging [#6096](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) will **increase** coverage by `6.49%`.
   > The diff coverage is `59.61%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-pinot/pull/6096/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz)](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #6096      +/-   ##
   ==========================================
   + Coverage   66.44%   72.94%   +6.49%     
   ==========================================
     Files        1075     1223     +148     
     Lines       54773    57731    +2958     
     Branches     8168     8518     +350     
   ==========================================
   + Hits        36396    42111    +5715     
   + Misses      15700    12864    -2836     
   - Partials     2677     2756      +79     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | #integration | `45.45% <48.65%> (?)` | |
   | #unittests | `64.03% <37.88%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `100.00% <ø> (ø)` | |
   | [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `52.83% <0.00%> (-13.84%)` | :arrow_down: |
   | [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `53.33% <0.00%> (-3.81%)` | :arrow_down: |
   | [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `44.44% <0.00%> (-4.40%)` | :arrow_down: |
   | [.../org/apache/pinot/client/ResultTableResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L1Jlc3VsdFRhYmxlUmVzdWx0U2V0LmphdmE=) | `24.00% <0.00%> (-10.29%)` | :arrow_down: |
   | [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `78.57% <ø> (+5.40%)` | :arrow_up: |
   | [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
   | [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `98.27% <ø> (-1.73%)` | :arrow_down: |
   | [.../pinot/common/function/DateTimePatternHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRGF0ZVRpbWVQYXR0ZXJuSGFuZGxlci5qYXZh) | `83.33% <ø> (ø)` | |
   | [...ot/common/function/FunctionDefinitionRegistry.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25EZWZpbml0aW9uUmVnaXN0cnkuamF2YQ==) | `88.88% <ø> (+44.44%)` | :arrow_up: |
   | ... and [960 more](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=footer). Last update [9929dad...d47729b](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] yupeng9 commented on pull request #6096: add upsert related configs

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#issuecomment-703194819


   > After some consideration, I still think it is better to wrap the primary key columns inside the `UpsertConfig` for the following reasons:
   > 
   > * The schema for the offline segment is re-constructed from the segment metadata, where primary key columns might not be maintained
   This is fine for now. As upsert applies to real-time segment only. In the future, I think we shall consider adding the primary ey concept to offline segment too
   > * For regular use cases, there is no primary key concept in Pinot. Putting it in `UpsertConfig` can keep all related configs at the same place
   > * For the upcoming JOIN feature, we should have another config for the in-memory lookup table, where we can put the primary key there. This 2 features cannot be applied to the same table, and I don't see the benefit of keeping this info inside the schema
   For fact-dim join, the primary key actually needs to be defined on the regular table.
   > 
   > Since this is the first feature using the primary key concept, I suggest putting all the related configs at the same place for easier management. If we find this as a common concept in the future, we can move it then.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #6096: add upsert related configs

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#issuecomment-703161952


   After some consideration, I still think it is better to wrap the primary key columns inside the `UpsertConfig` for the following reasons:
   - The schema for the offline segment is re-constructed from the segment metadata, where primary key columns might not be maintained
   - For regular use cases, there is no primary key concept in Pinot. Putting it in `UpsertConfig` can keep all related configs at the same place
   - For the upcoming JOIN feature, we should have another config for the in-memory lookup table, where we can put the primary key there. This 2 features cannot be applied to the same table, and I don't see the benefit of keeping this info inside the schema
   
   Since this is the first feature using the primary key concept, I suggest putting all the related configs at the same place for easier management. If we find this as a common concept in the future, we can move it then.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6096: add upsert related configs

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#discussion_r499173365



##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/data/Schema.java
##########
@@ -99,6 +102,14 @@ public void setSchemaName(String schemaName) {
     _schemaName = schemaName;
   }
 
+  public void setPrimaryKeyColumns(List<String> primaryKeyColumns) {
+    _primaryKeyColumns = primaryKeyColumns;
+  }
+
+  public List<String> getPrimaryKeyColumns() {

Review comment:
       (nit) put getter before setter for consistency

##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/data/Schema.java
##########
@@ -558,6 +576,11 @@ public SchemaBuilder addComplex(String name, DataType dataType) {
       return this;
     }
 
+    public SchemaBuilder addPrimaryKeyColumns(List<String> primaryKeyColumns) {

Review comment:
       ```suggestion
       public SchemaBuilder setPrimaryKeyColumns(List<String> primaryKeyColumns) {
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] yupeng9 commented on a change in pull request #6096: add upsert related configs

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#discussion_r499107830



##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/TableConfig.java
##########
@@ -265,4 +266,9 @@ public void setIngestionConfig(IngestionConfig ingestionConfig) {
   public void setTierConfigsList(List<TierConfig> tierConfigsList) {
     _tierConfigsList = tierConfigsList;
   }
+
+  @JsonIgnore
+  public UpsertConfig.Mode getUpsertMode() {

Review comment:
       yes, in the upcoming PRs




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org