You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/09/08 11:40:09 UTC

[GitHub] [doris] morrySnow opened a new pull request, #12483: [enhancement](Nereids) add all necessary PhysicalDistribute in CostAndEnforcerJob

morrySnow opened a new pull request, #12483:
URL: https://github.com/apache/doris/pull/12483

   # Proposed changes
   
   In an earlier PR #11976 , we add shuffle join and bucket shuffle support. But if join's right child's distribution spec satisfied join's require, we do not add distribute on right child. Instead of, do it in plan translator.
   It is hard to calculate accurate cost in this way, since we some distribute cost do not calculated.
   In this PR, we introduce a new shuffle type BUCKET, and change the way of add enforce to ensure all necessary distribute will be added in cost and enforcer job.
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: 
       - [ ] Yes
       - [ ] No
       - [ ] I don't know
   2. Has unit tests been added:
       - [ ] Yes
       - [ ] No
       - [ ] No Need
   3. Has document been added or modified:
       - [ ] Yes
       - [ ] No
       - [ ] No Need
   4. Does it need to update dependencies:
       - [ ] Yes
       - [ ] No
   5. Are there any changes that cannot be rolled back:
       - [ ] Yes (If Yes, please explain WHY)
       - [ ] No
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] englefly commented on a diff in pull request #12483: [enhancement](Nereids) add all necessary PhysicalDistribute on Join's child and update cost model

Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #12483:
URL: https://github.com/apache/doris/pull/12483#discussion_r975937935


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostCalculator.java:
##########
@@ -106,19 +112,47 @@ public CostEstimate visitPhysicalLocalQuickSort(
             return new CostEstimate(
                     childStatistics.computeSize(),
                     statistics.computeSize(),
-                    childStatistics.computeSize());
+                    0);
         }
 
         @Override
         public CostEstimate visitPhysicalDistribute(
                 PhysicalDistribute<? extends Plan> distribute, PlanContext context) {
-            StatsDeriveResult statistics = context.getStatisticsWithCheck();
             StatsDeriveResult childStatistics = context.getChildStatistics(0);
+            DistributionSpec spec = distribute.getDistributionSpec();
+            // shuffle
+            if (spec instanceof DistributionSpecHash) {
+                return new CostEstimate(
+                        childStatistics.computeSize(),
+                        0,
+                        childStatistics.computeSize());
+            }
 
+            // replicate
+            if (spec instanceof DistributionSpecReplicated) {
+                int beNumber = ConnectContext.get().getEnv().getClusterInfo().getBackendIds(true).size();
+                int instanceNumber = ConnectContext.get().getSessionVariable().getParallelExecInstanceNum();
+                beNumber = Math.max(1, beNumber);
+
+                return new CostEstimate(
+                        childStatistics.computeSize() * beNumber,

Review Comment:
   do not multiply by beNumber. In fact, for replicate mode, the beNumber is bigger, the workload is smaller.
   That means we need cost of a single BE node, not the cost of cluster.



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostCalculator.java:
##########
@@ -158,22 +192,10 @@ public CostEstimate visitPhysicalNestedLoopJoin(
 
             StatsDeriveResult leftStatistics = context.getChildStatistics(0);
             StatsDeriveResult rightStatistics = context.getChildStatistics(1);
-            List<Id> leftIds = context.getChildOutputIds(0);
-            List<Id> rightIds = context.getChildOutputIds(1);
-
-            // TODO: handle some case
-            // handle cross join, onClause is empty .....
-            if (nestedLoopJoin.getJoinType().isCrossJoin()) {
-                return new CostEstimate(
-                        leftStatistics.computeColumnSize(leftIds) + rightStatistics.computeColumnSize(rightIds),
-                        rightStatistics.computeColumnSize(rightIds),
-                        0);
-            }
 
-            // TODO: network 0?
             return new CostEstimate(
-                    (leftStatistics.computeColumnSize(leftIds) + rightStatistics.computeColumnSize(rightIds)) / 2,
-                    rightStatistics.computeColumnSize(rightIds),
+                    leftStatistics.computeSize() * rightStatistics.computeSize(),
+                    rightStatistics.computeSize(),

Review Comment:
   In my understanding, the memory is the size of two blocks, from left child and right child respectively.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] morrySnow merged pull request #12483: [enhancement](Nereids) add all necessary PhysicalDistribute on Join's child and update cost model

Posted by GitBox <gi...@apache.org>.
morrySnow merged PR #12483:
URL: https://github.com/apache/doris/pull/12483


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] jackwener commented on a diff in pull request #12483: [enhancement](Nereids) add all necessary PhysicalDistribute in CostAndEnforcerJob

Posted by GitBox <gi...@apache.org>.
jackwener commented on code in PR #12483:
URL: https://github.com/apache/doris/pull/12483#discussion_r965910325


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/CostAndEnforcerJob.java:
##########
@@ -184,33 +186,47 @@ public void execute() {
                 curTotalCost -= curNodeCost;
                 curNodeCost = CostCalculator.calculateCost(groupExpression);
                 curTotalCost += curNodeCost;
-                // record map { outputProperty -> outputProperty }, { ANY -> outputProperty },
-                recordPropertyAndCost(groupExpression, outputProperty, outputProperty, requestChildrenProperty);
-                recordPropertyAndCost(groupExpression, outputProperty, PhysicalProperties.ANY, requestChildrenProperty);
 
-                enforce(outputProperty, requestChildrenProperty);
+                // colocate join and bucket shuffle join cannot add enforce to satisfy required property.
+                if (enforce(outputProperty, requestChildrenProperty)) {
 
-                if (curTotalCost < context.getCostUpperBound()) {
-                    context.setCostUpperBound(curTotalCost);
+                    // record map { outputProperty -> outputProperty }, { ANY -> outputProperty },
+                    recordPropertyAndCost(groupExpression, outputProperty, outputProperty, requestChildrenProperty);
+                    recordPropertyAndCost(groupExpression, outputProperty, PhysicalProperties.ANY,
+                            requestChildrenProperty);

Review Comment:
   recordPropertyAndCost may be success instead of putting inside if.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #12483: [enhancement](Nereids) add all necessary PhysicalDistribute in CostAndEnforcerJob

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #12483:
URL: https://github.com/apache/doris/pull/12483#issuecomment-1240948526

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] morrySnow commented on pull request #12483: [enhancement](Nereids) add all necessary PhysicalDistribute in CostAndEnforcerJob

Posted by GitBox <gi...@apache.org>.
morrySnow commented on PR #12483:
URL: https://github.com/apache/doris/pull/12483#issuecomment-1241877026

   need to refactor whole enforce. need some time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] morrySnow commented on a diff in pull request #12483: [enhancement](Nereids) add all necessary PhysicalDistribute in CostAndEnforcerJob

Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #12483:
URL: https://github.com/apache/doris/pull/12483#discussion_r965934727


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/CostAndEnforcerJob.java:
##########
@@ -184,33 +186,47 @@ public void execute() {
                 curTotalCost -= curNodeCost;
                 curNodeCost = CostCalculator.calculateCost(groupExpression);
                 curTotalCost += curNodeCost;
-                // record map { outputProperty -> outputProperty }, { ANY -> outputProperty },
-                recordPropertyAndCost(groupExpression, outputProperty, outputProperty, requestChildrenProperty);
-                recordPropertyAndCost(groupExpression, outputProperty, PhysicalProperties.ANY, requestChildrenProperty);
 
-                enforce(outputProperty, requestChildrenProperty);
+                // colocate join and bucket shuffle join cannot add enforce to satisfy required property.
+                if (enforce(outputProperty, requestChildrenProperty)) {
 
-                if (curTotalCost < context.getCostUpperBound()) {
-                    context.setCostUpperBound(curTotalCost);
+                    // record map { outputProperty -> outputProperty }, { ANY -> outputProperty },
+                    recordPropertyAndCost(groupExpression, outputProperty, outputProperty, requestChildrenProperty);
+                    recordPropertyAndCost(groupExpression, outputProperty, PhysicalProperties.ANY,
+                            requestChildrenProperty);

Review Comment:
   enforce return false means this branch is not work



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #12483: [enhancement](Nereids) add all necessary PhysicalDistribute in CostAndEnforcerJob

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #12483:
URL: https://github.com/apache/doris/pull/12483#issuecomment-1240948573

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] morrySnow commented on a diff in pull request #12483: [enhancement](Nereids) add all necessary PhysicalDistribute on Join's child and update cost model

Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #12483:
URL: https://github.com/apache/doris/pull/12483#discussion_r975966586


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostCalculator.java:
##########
@@ -106,19 +112,47 @@ public CostEstimate visitPhysicalLocalQuickSort(
             return new CostEstimate(
                     childStatistics.computeSize(),
                     statistics.computeSize(),
-                    childStatistics.computeSize());
+                    0);
         }
 
         @Override
         public CostEstimate visitPhysicalDistribute(
                 PhysicalDistribute<? extends Plan> distribute, PlanContext context) {
-            StatsDeriveResult statistics = context.getStatisticsWithCheck();
             StatsDeriveResult childStatistics = context.getChildStatistics(0);
+            DistributionSpec spec = distribute.getDistributionSpec();
+            // shuffle
+            if (spec instanceof DistributionSpecHash) {
+                return new CostEstimate(
+                        childStatistics.computeSize(),
+                        0,
+                        childStatistics.computeSize());
+            }
 
+            // replicate
+            if (spec instanceof DistributionSpecReplicated) {
+                int beNumber = ConnectContext.get().getEnv().getClusterInfo().getBackendIds(true).size();
+                int instanceNumber = ConnectContext.get().getSessionVariable().getParallelExecInstanceNum();
+                beNumber = Math.max(1, beNumber);
+
+                return new CostEstimate(
+                        childStatistics.computeSize() * beNumber,

Review Comment:
   This PR aim to get a correct plan of colocate join and bucket shuffle join. Good cost model is not including. We need to adjust it finely after this PR and #12767
    



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostCalculator.java:
##########
@@ -158,22 +192,10 @@ public CostEstimate visitPhysicalNestedLoopJoin(
 
             StatsDeriveResult leftStatistics = context.getChildStatistics(0);
             StatsDeriveResult rightStatistics = context.getChildStatistics(1);
-            List<Id> leftIds = context.getChildOutputIds(0);
-            List<Id> rightIds = context.getChildOutputIds(1);
-
-            // TODO: handle some case
-            // handle cross join, onClause is empty .....
-            if (nestedLoopJoin.getJoinType().isCrossJoin()) {
-                return new CostEstimate(
-                        leftStatistics.computeColumnSize(leftIds) + rightStatistics.computeColumnSize(rightIds),
-                        rightStatistics.computeColumnSize(rightIds),
-                        0);
-            }
 
-            // TODO: network 0?
             return new CostEstimate(
-                    (leftStatistics.computeColumnSize(leftIds) + rightStatistics.computeColumnSize(rightIds)) / 2,
-                    rightStatistics.computeColumnSize(rightIds),
+                    leftStatistics.computeSize() * rightStatistics.computeSize(),
+                    rightStatistics.computeSize(),

Review Comment:
   This PR aim to get a correct plan of colocate join and bucket shuffle join. Good cost model is not including. We need to adjust it finely after this PR and https://github.com/apache/doris/pull/12767



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org