You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/06/09 14:19:50 UTC
[GitHub] [incubator-doris] morrySnow opened a new pull request, #10045: [fix] (planner) produce wrong result when use bucket shuffle join with colocate left table
morrySnow opened a new pull request, #10045:
URL: https://github.com/apache/incubator-doris/pull/10045
# Proposed changes
change the bucket num compute method in bucket shuffle join
## Problem Summary:
When plan bucket shuffle join, we need to know left table bucket number.
Currently, we use tablet number directly based on the assumption that left table has only one partition.
But, when left table is colocated table, it could have more than one partition.
In this case, some data in right table will be dropped incorrectly and produce wrong result for query.
reproduce could follow regression test in PR.
## Checklist(Required)
1. Does it affect the original behavior: No
2. Has unit tests been added: No Need
3. Has document been added or modified: No Need
4. Does it need to update dependencies: No
5. Are there any changes that cannot be rolled back: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #10045: [fix] (planner) produce wrong result when use bucket shuffle join with colocate left table
Posted by GitBox <gi...@apache.org>.
morningman merged PR #10045:
URL: https://github.com/apache/incubator-doris/pull/10045
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morrySnow commented on a diff in pull request #10045: [fix] (planner) produce wrong result when use bucket shuffle join with colocate left table
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #10045:
URL: https://github.com/apache/incubator-doris/pull/10045#discussion_r893714076
##########
fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java:
##########
@@ -1850,9 +1850,19 @@ private void getExecHostPortForFragmentIDAndBucketSeq(TScanRangeLocations seqLoc
private void computeScanRangeAssignmentByBucket(
final OlapScanNode scanNode, ImmutableMap<Long, Backend> idToBackend, Map<TNetworkAddress, Long> addressToBackendID) throws Exception {
if (!fragmentIdToSeqToAddressMap.containsKey(scanNode.getFragmentId())) {
- // The bucket shuffle join only hit when the partition is one. so the totalTabletsNum is all tablet of
- // one hit partition. can be the right bucket num in bucket shuffle join
- fragmentIdToBucketNumMap.put(scanNode.getFragmentId(), (int) scanNode.getTotalTabletsNum());
+ // In bucket shuffle join, we have 2 situation.
+ // 1. Only one partition: in this case, we use scanNode.getTotalTabletsNum() to get the right bucket num
+ // because when table turn on dynamic partition, the bucket number in default distribution info
+ // is not correct.
+ // 2. Table is colocated: in this case, table could have more than one partition, but all partition's
+ // bucket number must be same, so we use default bucket num is ok.
+ int bucketNum = 0;
+ if (scanNode.getOlapTable().isColocateTable()) {
Review Comment:
already handled in [DistributedPlanner ](https://github.com/apache/incubator-doris/blob/e701c057dc675babe7a859e151287fea7cd5c850/fe/fe-core/src/main/java/org/apache/doris/planner/DistributedPlanner.java#L618)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #10045: [fix] (planner) produce wrong result when use bucket shuffle join with colocate left table
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #10045:
URL: https://github.com/apache/incubator-doris/pull/10045#issuecomment-1151812589
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #10045: [fix] (planner) produce wrong result when use bucket shuffle join with colocate left table
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #10045:
URL: https://github.com/apache/incubator-doris/pull/10045#issuecomment-1151812615
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a diff in pull request #10045: [fix] (planner) produce wrong result when use bucket shuffle join with colocate left table
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #10045:
URL: https://github.com/apache/incubator-doris/pull/10045#discussion_r893589163
##########
fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java:
##########
@@ -1850,9 +1850,19 @@ private void getExecHostPortForFragmentIDAndBucketSeq(TScanRangeLocations seqLoc
private void computeScanRangeAssignmentByBucket(
final OlapScanNode scanNode, ImmutableMap<Long, Backend> idToBackend, Map<TNetworkAddress, Long> addressToBackendID) throws Exception {
if (!fragmentIdToSeqToAddressMap.containsKey(scanNode.getFragmentId())) {
- // The bucket shuffle join only hit when the partition is one. so the totalTabletsNum is all tablet of
- // one hit partition. can be the right bucket num in bucket shuffle join
- fragmentIdToBucketNumMap.put(scanNode.getFragmentId(), (int) scanNode.getTotalTabletsNum());
+ // In bucket shuffle join, we have 2 situation.
+ // 1. Only one partition: in this case, we use scanNode.getTotalTabletsNum() to get the right bucket num
+ // because when table turn on dynamic partition, the bucket number in default distribution info
+ // is not correct.
+ // 2. Table is colocated: in this case, table could have more than one partition, but all partition's
+ // bucket number must be same, so we use default bucket num is ok.
+ int bucketNum = 0;
+ if (scanNode.getOlapTable().isColocateTable()) {
Review Comment:
What if the colocate group is unstable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org