You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "yuzelin (via GitHub)" <gi...@apache.org> on 2024/04/09 07:20:33 UTC
[PR] [flink] Add merge-engine check when executing row level batch update and delete [paimon]
yuzelin opened a new pull request, #3181:
URL: https://github.com/apache/paimon/pull/3181
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
1. partial-update and aggregate should not accept batch update because there is no UB and the aggregate result will be wrong.
2. It's not very clear about how to handle row level batch delete with partial update currently. Doris just delete the record. But it's not easy to delete the record because #3128 and #3139.
<!-- Linking this pull request to the issue -->
Linked issue: close #xxx
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
Support batch update merge engines: `deduplicate` and `first-row`.
Support batch delete merge engines: `deduplicate`.
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Add merge-engine check when executing row level batch update and delete [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #3181:
URL: https://github.com/apache/paimon/pull/3181#issuecomment-2054320226
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Add merge-engine check when executing row level batch update and delete [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #3181:
URL: https://github.com/apache/paimon/pull/3181#discussion_r1557300739
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -1692,20 +1692,28 @@ public boolean deletionVectorsEnabled() {
/** Specifies the merge engine for table with primary key. */
public enum MergeEngine implements DescribedEnum {
- DEDUPLICATE("deduplicate", "De-duplicate and keep the last row."),
+ DEDUPLICATE("deduplicate", "De-duplicate and keep the last row.", true, true),
- PARTIAL_UPDATE("partial-update", "Partial update non-null fields."),
+ PARTIAL_UPDATE("partial-update", "Partial update non-null fields.", false, false),
- AGGREGATE("aggregation", "Aggregate fields with same primary key."),
+ AGGREGATE("aggregation", "Aggregate fields with same primary key.", false, false),
- FIRST_ROW("first-row", "De-duplicate and keep the first row.");
+ FIRST_ROW("first-row", "De-duplicate and keep the first row.", true, false);
private final String value;
private final String description;
-
- MergeEngine(String value, String description) {
+ private final boolean supportBatchUpdate;
Review Comment:
Please do not modify this, this is public api, we don't need to expose here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Add merge-engine check when executing row level batch update and delete [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #3181:
URL: https://github.com/apache/paimon/pull/3181
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Add merge-engine check when executing row level batch update and delete [paimon]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #3181:
URL: https://github.com/apache/paimon/pull/3181#discussion_r1557202999
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -1692,20 +1692,28 @@ public boolean deletionVectorsEnabled() {
/** Specifies the merge engine for table with primary key. */
public enum MergeEngine implements DescribedEnum {
- DEDUPLICATE("deduplicate", "De-duplicate and keep the last row."),
+ DEDUPLICATE("deduplicate", "De-duplicate and keep the last row.", true, true),
- PARTIAL_UPDATE("partial-update", "Partial update non-null fields."),
+ PARTIAL_UPDATE("partial-update", "Partial update non-null fields.", false, false),
Review Comment:
When using partial-update and no col contains agg, update is theoretically supported
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Add merge-engine check when executing row level batch update and delete [paimon]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #3181:
URL: https://github.com/apache/paimon/pull/3181#discussion_r1557101892
##########
paimon-common/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -1692,20 +1692,28 @@ public boolean deletionVectorsEnabled() {
/** Specifies the merge engine for table with primary key. */
public enum MergeEngine implements DescribedEnum {
- DEDUPLICATE("deduplicate", "De-duplicate and keep the last row."),
+ DEDUPLICATE("deduplicate", "De-duplicate and keep the last row.", true, true),
- PARTIAL_UPDATE("partial-update", "Partial update non-null fields."),
+ PARTIAL_UPDATE("partial-update", "Partial update non-null fields.", false, false),
- AGGREGATE("aggregation", "Aggregate fields with same primary key."),
+ AGGREGATE("aggregation", "Aggregate fields with same primary key.", false, false),
- FIRST_ROW("first-row", "De-duplicate and keep the first row.");
+ FIRST_ROW("first-row", "De-duplicate and keep the first row.", true, false);
Review Comment:
first-row does not need to support update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org