You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/10/02 20:24:36 UTC
[GitHub] [incubator-pinot] yupeng9 opened a new pull request #6096: add upsert related configs
yupeng9 opened a new pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096
## Description
Part of a series of PRs for https://github.com/apache/incubator-pinot/issues/4261
Check this [doc](https://docs.google.com/document/d/1qljEMndPMxbbKtjlVn9mn2toz7Qrk0TGQsHLfI--7h8/edit#heading=h.lsfmyoyyxtgt) out for the new design
- Added the primary key definition in the schema
- Changed the upsert config in table config. For the first phase, only support the full update mode.
Does this PR otherwise need attention when creating release notes? Things to consider:
- New configuration options
configurations related to upsert
If you have a series of commits adding or enabling a feature, then
add this section only in final commit that marks the feature completed.
Refer to earlier release notes to see examples of text
## Documentation
If you have introduced a new feature or configuration, please add it to the documentation as well.
See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter commented on pull request #6096: add upsert related configs
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#issuecomment-703198725
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=h1) Report
> Merging [#6096](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) will **decrease** coverage by `20.98%`.
> The diff coverage is `48.65%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6096 +/- ##
===========================================
- Coverage 66.44% 45.45% -20.99%
===========================================
Files 1075 1223 +148
Lines 54773 57731 +2958
Branches 8168 8518 +350
===========================================
- Hits 36396 26244 -10152
- Misses 15700 29351 +13651
+ Partials 2677 2136 -541
```
| Flag | Coverage Δ | |
|---|---|---|
| #integration | `45.45% <48.65%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `100.00% <ø> (ø)` | |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `52.83% <0.00%> (-13.84%)` | :arrow_down: |
| [...org/apache/pinot/broker/queryquota/HitCounter.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9IaXRDb3VudGVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...che/pinot/broker/queryquota/MaxHitRateTracker.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9NYXhIaXRSYXRlVHJhY2tlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ache/pinot/broker/queryquota/QueryQuotaEntity.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9RdWVyeVF1b3RhRW50aXR5LmphdmE=) | `0.00% <0.00%> (-50.00%)` | :arrow_down: |
| [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `26.66% <0.00%> (-30.48%)` | :arrow_down: |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `22.22% <0.00%> (-26.62%)` | :arrow_down: |
| [.../org/apache/pinot/client/ResultTableResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L1Jlc3VsdFRhYmxlUmVzdWx0U2V0LmphdmE=) | `24.00% <0.00%> (-10.29%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `64.28% <ø> (-8.89%)` | :arrow_down: |
| [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
| ... and [1189 more](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=footer). Last update [9929dad...d47729b](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 commented on pull request #6096: add upsert related configs
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#issuecomment-703188624
I don't think so. Strictly speaking, primary key is a constraint defined on the schema, just like other constraints over the schema such as a column is `NOT NULL` or a column is `UNIQUE`. This is well defined in the database literature.
If the consideration is to limit its usage, then we can add a checker to ensure it's defined only when Upsert mode is not `NONE`.
However, for cleaniness (and also the ease of future development and structuring), the constraints shall be defined as part of the schema.
> After some consideration, I still think it is better to wrap the primary key columns inside the `UpsertConfig` for the following reasons:
>
> * The schema for the offline segment is re-constructed from the segment metadata, where primary key columns might not be maintained
> * For regular use cases, there is no primary key concept in Pinot. Putting it in `UpsertConfig` can keep all related configs at the same place
> * For the upcoming JOIN feature, we should have another config for the in-memory lookup table, where we can put the primary key there. This 2 features cannot be applied to the same table, and I don't see the benefit of keeping this info inside the schema
>
> Since this is the first feature using the primary key concept, I suggest putting all the related configs at the same place for easier management. If we find this as a common concept in the future, we can move it then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 commented on a change in pull request #6096: add upsert related configs
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#discussion_r499107805
##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/UpsertConfig.java
##########
@@ -27,44 +27,22 @@
public class UpsertConfig extends BaseJsonConfig {
- // names of the columns that used as primary keys of an upsert table
- private final List<String> _primaryKeyColumns;
- // name of the column that we are going to store the offset value to
- private final String _offsetColumn;
- // name of the virtual column that we are going to store the validFrom value to
- private final String _validFromColumn;
- // name of the virtual column that we are going to store the validUntil value to
- private final String _validUntilColumn;
+ public static final String MODE_KEY = "mode";
- @JsonCreator
- public UpsertConfig(@JsonProperty(value = "primaryKeyColumns") List<String> primaryKeyColumns,
- @JsonProperty(value = "offsetColumn") String offsetColumn,
- @JsonProperty(value = "validFromColumn") String validFromColumn,
- @JsonProperty(value = "validUntilColumn") String validUntilColumn) {
- Preconditions.checkArgument(primaryKeyColumns != null && primaryKeyColumns.size() == 1,
- "'primaryKeyColumns' must be configured with exact one column");
- Preconditions.checkArgument(StringUtils.isNotEmpty(offsetColumn), "'offsetColumn' must be configured");
- Preconditions.checkArgument(StringUtils.isNotEmpty(validFromColumn), "'validFromColumn' must be configured");
- Preconditions.checkArgument(StringUtils.isNotEmpty(validUntilColumn), "'validUntilColumn' must be configured");
- _primaryKeyColumns = primaryKeyColumns;
- _offsetColumn = offsetColumn;
- _validFromColumn = validFromColumn;
- _validUntilColumn = validUntilColumn;
+ public enum Mode {
+ FULL, PARTIAL, NONE
Review comment:
In the implementation, it's more clearer (and easier) to have a value to represent that upsert is disabled. Also, when UpsertConfig is null, it implies the mode is `NONE`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6096: add upsert related configs
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#discussion_r499105212
##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/UpsertConfig.java
##########
@@ -27,44 +27,22 @@
public class UpsertConfig extends BaseJsonConfig {
- // names of the columns that used as primary keys of an upsert table
- private final List<String> _primaryKeyColumns;
- // name of the column that we are going to store the offset value to
- private final String _offsetColumn;
- // name of the virtual column that we are going to store the validFrom value to
- private final String _validFromColumn;
- // name of the virtual column that we are going to store the validUntil value to
- private final String _validUntilColumn;
+ public static final String MODE_KEY = "mode";
- @JsonCreator
- public UpsertConfig(@JsonProperty(value = "primaryKeyColumns") List<String> primaryKeyColumns,
- @JsonProperty(value = "offsetColumn") String offsetColumn,
- @JsonProperty(value = "validFromColumn") String validFromColumn,
- @JsonProperty(value = "validUntilColumn") String validUntilColumn) {
- Preconditions.checkArgument(primaryKeyColumns != null && primaryKeyColumns.size() == 1,
- "'primaryKeyColumns' must be configured with exact one column");
- Preconditions.checkArgument(StringUtils.isNotEmpty(offsetColumn), "'offsetColumn' must be configured");
- Preconditions.checkArgument(StringUtils.isNotEmpty(validFromColumn), "'validFromColumn' must be configured");
- Preconditions.checkArgument(StringUtils.isNotEmpty(validUntilColumn), "'validUntilColumn' must be configured");
- _primaryKeyColumns = primaryKeyColumns;
- _offsetColumn = offsetColumn;
- _validFromColumn = validFromColumn;
- _validUntilColumn = validUntilColumn;
+ public enum Mode {
+ FULL, PARTIAL, NONE
Review comment:
Recommend not putting `NONE` as a mode. Upsert is disabled when the `UpsertConfig` is `null` in the table config (more intuitive IMO)
##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/data/Schema.java
##########
@@ -66,6 +66,9 @@
private TimeFieldSpec _timeFieldSpec;
private final List<DateTimeFieldSpec> _dateTimeFieldSpecs = new ArrayList<>();
private final List<ComplexFieldSpec> _complexFieldSpecs = new ArrayList<>();
+ // names of the columns that used as primary keys
+ // TODO: add validation checks like duplicate columns and use of time column
+ private List<String> _primaryKeyColumns;
Review comment:
This should not be part of the schema, but the upsert config
##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/TableConfig.java
##########
@@ -265,4 +266,9 @@ public void setIngestionConfig(IngestionConfig ingestionConfig) {
public void setTierConfigsList(List<TierConfig> tierConfigsList) {
_tierConfigsList = tierConfigsList;
}
+
+ @JsonIgnore
+ public UpsertConfig.Mode getUpsertMode() {
Review comment:
Not sure if this method is needed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 commented on a change in pull request #6096: add upsert related configs
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#discussion_r499107725
##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/data/Schema.java
##########
@@ -66,6 +66,9 @@
private TimeFieldSpec _timeFieldSpec;
private final List<DateTimeFieldSpec> _dateTimeFieldSpecs = new ArrayList<>();
private final List<ComplexFieldSpec> _complexFieldSpecs = new ArrayList<>();
+ // names of the columns that used as primary keys
+ // TODO: add validation checks like duplicate columns and use of time column
+ private List<String> _primaryKeyColumns;
Review comment:
Primary key is an inherent attribute of the schema. And its usage is beyond the upsert. For example, it's also used in the join feature which is happening in parallel.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 edited a comment on pull request #6096: add upsert related configs
Posted by GitBox <gi...@apache.org>.
yupeng9 edited a comment on pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#issuecomment-703194819
> After some consideration, I still think it is better to wrap the primary key columns inside the `UpsertConfig` for the following reasons:
>
> * The schema for the offline segment is re-constructed from the segment metadata, where primary key columns might not be maintained
This is fine for now. As upsert applies to real-time segment only. In the future, I think we shall consider adding the primary ey concept to offline segment too
> * For regular use cases, there is no primary key concept in Pinot. Putting it in `UpsertConfig` can keep all related configs at the same place
> * For the upcoming JOIN feature, we should have another config for the in-memory lookup table, where we can put the primary key there. This 2 features cannot be applied to the same table, and I don't see the benefit of keeping this info inside the schema
For fact-dim join, the primary key actually needs to be defined on the regular table.
>
> Since this is the first feature using the primary key concept, I suggest putting all the related configs at the same place for easier management. If we find this as a common concept in the future, we can move it then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 commented on pull request #6096: add upsert related configs
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#issuecomment-703195721
@Jackie-Jiang thanks for taking a look. comments addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #6096: add upsert related configs
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #6096: add upsert related configs
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#issuecomment-703198725
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=h1) Report
> Merging [#6096](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) will **increase** coverage by `6.49%`.
> The diff coverage is `59.61%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6096 +/- ##
==========================================
+ Coverage 66.44% 72.94% +6.49%
==========================================
Files 1075 1223 +148
Lines 54773 57731 +2958
Branches 8168 8518 +350
==========================================
+ Hits 36396 42111 +5715
+ Misses 15700 12864 -2836
- Partials 2677 2756 +79
```
| Flag | Coverage Δ | |
|---|---|---|
| #integration | `45.45% <48.65%> (?)` | |
| #unittests | `64.03% <37.88%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `100.00% <ø> (ø)` | |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `52.83% <0.00%> (-13.84%)` | :arrow_down: |
| [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `53.33% <0.00%> (-3.81%)` | :arrow_down: |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `44.44% <0.00%> (-4.40%)` | :arrow_down: |
| [.../org/apache/pinot/client/ResultTableResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L1Jlc3VsdFRhYmxlUmVzdWx0U2V0LmphdmE=) | `24.00% <0.00%> (-10.29%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `78.57% <ø> (+5.40%)` | :arrow_up: |
| [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
| [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `98.27% <ø> (-1.73%)` | :arrow_down: |
| [.../pinot/common/function/DateTimePatternHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRGF0ZVRpbWVQYXR0ZXJuSGFuZGxlci5qYXZh) | `83.33% <ø> (ø)` | |
| [...ot/common/function/FunctionDefinitionRegistry.java](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25EZWZpbml0aW9uUmVnaXN0cnkuamF2YQ==) | `88.88% <ø> (+44.44%)` | :arrow_up: |
| ... and [960 more](https://codecov.io/gh/apache/incubator-pinot/pull/6096/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=footer). Last update [9929dad...d47729b](https://codecov.io/gh/apache/incubator-pinot/pull/6096?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 commented on pull request #6096: add upsert related configs
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#issuecomment-703194819
> After some consideration, I still think it is better to wrap the primary key columns inside the `UpsertConfig` for the following reasons:
>
> * The schema for the offline segment is re-constructed from the segment metadata, where primary key columns might not be maintained
This is fine for now. As upsert applies to real-time segment only. In the future, I think we shall consider adding the primary ey concept to offline segment too
> * For regular use cases, there is no primary key concept in Pinot. Putting it in `UpsertConfig` can keep all related configs at the same place
> * For the upcoming JOIN feature, we should have another config for the in-memory lookup table, where we can put the primary key there. This 2 features cannot be applied to the same table, and I don't see the benefit of keeping this info inside the schema
For fact-dim join, the primary key actually needs to be defined on the regular table.
>
> Since this is the first feature using the primary key concept, I suggest putting all the related configs at the same place for easier management. If we find this as a common concept in the future, we can move it then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #6096: add upsert related configs
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#issuecomment-703161952
After some consideration, I still think it is better to wrap the primary key columns inside the `UpsertConfig` for the following reasons:
- The schema for the offline segment is re-constructed from the segment metadata, where primary key columns might not be maintained
- For regular use cases, there is no primary key concept in Pinot. Putting it in `UpsertConfig` can keep all related configs at the same place
- For the upcoming JOIN feature, we should have another config for the in-memory lookup table, where we can put the primary key there. This 2 features cannot be applied to the same table, and I don't see the benefit of keeping this info inside the schema
Since this is the first feature using the primary key concept, I suggest putting all the related configs at the same place for easier management. If we find this as a common concept in the future, we can move it then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6096: add upsert related configs
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#discussion_r499173365
##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/data/Schema.java
##########
@@ -99,6 +102,14 @@ public void setSchemaName(String schemaName) {
_schemaName = schemaName;
}
+ public void setPrimaryKeyColumns(List<String> primaryKeyColumns) {
+ _primaryKeyColumns = primaryKeyColumns;
+ }
+
+ public List<String> getPrimaryKeyColumns() {
Review comment:
(nit) put getter before setter for consistency
##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/data/Schema.java
##########
@@ -558,6 +576,11 @@ public SchemaBuilder addComplex(String name, DataType dataType) {
return this;
}
+ public SchemaBuilder addPrimaryKeyColumns(List<String> primaryKeyColumns) {
Review comment:
```suggestion
public SchemaBuilder setPrimaryKeyColumns(List<String> primaryKeyColumns) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 commented on a change in pull request #6096: add upsert related configs
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #6096:
URL: https://github.com/apache/incubator-pinot/pull/6096#discussion_r499107830
##########
File path: pinot-spi/src/main/java/org/apache/pinot/spi/config/table/TableConfig.java
##########
@@ -265,4 +266,9 @@ public void setIngestionConfig(IngestionConfig ingestionConfig) {
public void setTierConfigsList(List<TierConfig> tierConfigsList) {
_tierConfigsList = tierConfigsList;
}
+
+ @JsonIgnore
+ public UpsertConfig.Mode getUpsertMode() {
Review comment:
yes, in the upcoming PRs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org