You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/10/20 17:00:19 UTC
[GitHub] [doris] AshinGau opened a new pull request, #13533: [improvement](planner) support delete from partitioned table without …
AshinGau opened a new pull request, #13533:
URL: https://github.com/apache/doris/pull/13533
…partition specified
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [x] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [x] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [x] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [x] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [x] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13533: [improvement](planner) support delete from partitioned table without partition specified
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #13533:
URL: https://github.com/apache/doris/pull/13533#issuecomment-1292908811
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 38.47 seconds
load time: 568 seconds
storage size: 17154644735 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221027110926_clickbench_pr_34515.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman merged pull request #13533: [improvement](planner) support delete from partitioned table without partition specified
Posted by GitBox <gi...@apache.org>.
morningman merged PR #13533:
URL: https://github.com/apache/doris/pull/13533
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13533: [improvement](planner) support delete from partitioned table without partition specified
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #13533:
URL: https://github.com/apache/doris/pull/13533#issuecomment-1286208196
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 41.17 seconds
load time: 571 seconds
storage size: 17154602793 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221021060410_clickbench_pr_32258.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13533: [improvement](planner) support delete from partitioned table without partition specified
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13533:
URL: https://github.com/apache/doris/pull/13533#issuecomment-1292996034
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13533: [improvement](planner) support delete from partitioned table without partition specified
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #13533:
URL: https://github.com/apache/doris/pull/13533#issuecomment-1286837707
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 40.23 seconds
load time: 565 seconds
storage size: 17154577937 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221021193408_clickbench_pr_32517.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #13533: [improvement](planner) support delete from partitioned table without partition specified
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #13533:
URL: https://github.com/apache/doris/pull/13533#discussion_r1002727749
##########
fe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java:
##########
@@ -162,14 +173,46 @@ public void process(DeleteStmt stmt) throws DdlException, QueryStateException {
}
if (noPartitionSpecified) {
+ // Try to get selected partitions if no partition specified in delete statement
+ // Use PartitionPruner to generate the select partitions
if (olapTable.getPartitionInfo().getType() == PartitionType.RANGE
|| olapTable.getPartitionInfo().getType() == PartitionType.LIST) {
- if (!ConnectContext.get().getSessionVariable().isDeleteWithoutPartition()) {
- throw new DdlException("This is a range or list partitioned table."
- + " You should specify partition in delete stmt,"
- + " or set delete_without_partition to true");
+ Set<String> partitionColumnNameSet = Utils.execWithReturnVal(
+ olapTable::getPartitionColumnNames);
+ Map<String, ColumnRange> columnNameToRange = Maps.newHashMap();
+ for (String colName : partitionColumnNameSet) {
+ ColumnRange columnRange = createColumnRange(colName, conditions);
+ // Not all partition columns are involved in predicate conditions
+ if (columnRange != null) {
+ columnNameToRange.put(colName, columnRange);
+ }
+ }
+
+ Collection<Long> selectedPartitionId = null;
+ if (!columnNameToRange.isEmpty()) {
+ PartitionInfo partitionInfo = olapTable.getPartitionInfo();
+ Map<Long, PartitionItem> keyItemMap = partitionInfo.getIdToItem(false);
+ PartitionPruner pruner = olapTable.getPartitionInfo().getType() == PartitionType.RANGE
+ ? new RangePartitionPrunerV2(keyItemMap, partitionInfo.getPartitionColumns(),
+ columnNameToRange)
+ : new ListPartitionPrunerV2(keyItemMap, partitionInfo.getPartitionColumns(),
+ columnNameToRange);
+ selectedPartitionId = Utils.execWithReturnVal(pruner::prune);
Review Comment:
Call `prune` directly?
##########
fe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java:
##########
@@ -162,14 +173,46 @@ public void process(DeleteStmt stmt) throws DdlException, QueryStateException {
}
if (noPartitionSpecified) {
+ // Try to get selected partitions if no partition specified in delete statement
+ // Use PartitionPruner to generate the select partitions
if (olapTable.getPartitionInfo().getType() == PartitionType.RANGE
|| olapTable.getPartitionInfo().getType() == PartitionType.LIST) {
- if (!ConnectContext.get().getSessionVariable().isDeleteWithoutPartition()) {
- throw new DdlException("This is a range or list partitioned table."
- + " You should specify partition in delete stmt,"
- + " or set delete_without_partition to true");
+ Set<String> partitionColumnNameSet = Utils.execWithReturnVal(
Review Comment:
Why not just call `olapTable.getPartitionColumnNames()`?
I think we should not eliminate those exceptions.
##########
fe/fe-core/src/main/java/org/apache/doris/load/DeleteHandler.java:
##########
@@ -369,6 +412,77 @@ public void process(DeleteStmt stmt) throws DdlException, QueryStateException {
}
}
+ // Return null if there is no filter for the partition column
+ private ColumnRange createColumnRange(String colName, List<Predicate> conditions) {
+ ColumnRange result = ColumnRange.create();
+ boolean hasRange = false;
+ for (Predicate predicate : conditions) {
+ List<Range<ColumnBound>> bounds = createColumnRange(colName, predicate);
+ if (bounds != null) {
+ hasRange = true;
+ result.intersect(bounds);
+ }
+ }
+ if (hasRange) {
+ return result;
+ } else {
+ return null;
+ }
+ }
+
+ // Return null if the condition is not related to the partition column,
+ // or the operator is not supported.
+ private List<Range<ColumnBound>> createColumnRange(String colName, Predicate condition) {
+ List<Range<ColumnBound>> result = Lists.newLinkedList();
+ if (condition instanceof BinaryPredicate) {
+ BinaryPredicate binaryPredicate = (BinaryPredicate) condition;
+ String columnName = ((SlotRef) binaryPredicate.getChild(0)).getColumnName();
+ if (!colName.equals(columnName)) {
Review Comment:
```suggestion
if (!colName.equalsIgnoreCase(columnName)) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13533: [improvement](planner) support delete from partitioned table without partition specified
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13533:
URL: https://github.com/apache/doris/pull/13533#issuecomment-1292996060
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org