You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/11/22 09:01:09 UTC
[GitHub] [doris] Kikyou1997 opened a new pull request, #14490: [ehancement](nereids) eliminate project in the post process phase
Kikyou1997 opened a new pull request, #14490:
URL: https://github.com/apache/doris/pull/14490
# Proposed changes
## Problem summary
Remove those projects that used for column pruning only and don't do any expression calculation, So that we could avoid some redundant data copy in do_projection of BE side.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #14490: [ehancement](nereids) eliminate project in the post process phase
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #14490:
URL: https://github.com/apache/doris/pull/14490#discussion_r1032136253
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -1246,4 +1318,44 @@ private List<Integer> getSlotIdList(TupleDescriptor tupleDescriptor) {
.map(slot -> slot.getId().asInt())
.collect(ImmutableList.toImmutableList());
}
+
+ private boolean isUnnecessaryProject(PhysicalProject project) {
+ // The project list for agg is always needed,since tuple of agg contains the slots used by group by expr
+ if (project.child() instanceof PhysicalAggregate) {
Review Comment:
how about
```sql
SELECT c1 AS a1, c2 AS a2 FROM (SELECT c1, COUNT(c3) AS c2 FROM t1 GROUP BY c1) t2
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -135,6 +136,13 @@ public PlanFragment translatePlan(PhysicalPlan physicalPlan, PlanTranslatorConte
rootFragment = exchangeToMergeFragment(rootFragment, context);
}
List<Expr> outputExprs = Lists.newArrayList();
+ if (physicalPlan instanceof PhysicalProject) {
+ PhysicalProject project = (PhysicalProject) physicalPlan;
+ if (isUnnecessaryProject(project)) {
+ List<Slot> slotReferences = removeAlias(project);
+ physicalPlan = (PhysicalPlan) physicalPlan.child(0).withOutput(slotReferences);
Review Comment:
no need to update child's output. generate outputExprs according project's output's order is enough
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -675,28 +683,53 @@ public PlanFragment visitPhysicalHashJoin(
if (hashJoin.getOtherJoinConjuncts().isEmpty()
&& (joinType == JoinType.LEFT_ANTI_JOIN || joinType == JoinType.LEFT_SEMI_JOIN)) {
for (SlotDescriptor leftSlotDescriptor : leftSlotDescriptors) {
+ if (!leftSlotDescriptor.isMaterialized()) {
+ continue;
+ }
SlotReference sf = leftChildOutputMap.get(context.findExprId(leftSlotDescriptor.getId()));
+ if (sf == null) {
+ continue;
+ }
Review Comment:
add this because we remove project on olapScan? chould we add a method to TupleDescriptor to remove slot from tuple descriptor.
`updateChildSlotsMaterialization` works like remove slot instead of set slot to Materialized.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -901,42 +958,57 @@ public PlanFragment visitPhysicalProject(PhysicalProject<? extends Plan> project
}
}
PlanFragment inputFragment = project.child(0).accept(this, context);
-
List<Expr> execExprList = project.getProjects()
.stream()
.map(e -> ExpressionTranslator.translate(e, context))
.collect(Collectors.toList());
// TODO: fix the project alias of an aliased relation.
- List<Slot> slotList = project.getOutput();
- TupleDescriptor tupleDescriptor = generateTupleDesc(slotList, null, context);
+
PlanNode inputPlanNode = inputFragment.getPlanRoot();
+ List<Slot> slotList = project.getOutput();
+ if (!hasExprCalc(project)) {
+ List<NamedExpression> namedExpressions = project.getProjects();
+ for (int i = 0; i < namedExpressions.size(); i++) {
+ NamedExpression n = namedExpressions.get(i);
+ for (Expression e : n.children()) {
+ SlotReference slotReference = (SlotReference) e;
+ SlotRef slotRef = context.findSlotRef(slotReference.getExprId());
+ context.addExprIdSlotRefPair(slotList.get(i).getExprId(), slotRef);
+ }
+ }
+ } else {
+ if (!(inputPlanNode instanceof OlapScanNode && !hasExprCalc(project))) {
+ TupleDescriptor tupleDescriptor = generateTupleDesc(slotList, null, context);
+ inputPlanNode.setProjectList(execExprList);
+ inputPlanNode.setOutputTupleDesc(tupleDescriptor);
+ }
+ }
// For hash join node, use vSrcToOutputSMap to describe the expression calculation, use
// vIntermediateTupleDescList as input, and set vOutputTupleDesc as the final output.
// TODO: HashJoinNode's be implementation is not support projection yet, remove this after when supported.
if (inputPlanNode instanceof HashJoinNode) {
Review Comment:
use `inputPlanNode instanceOf JoinNodeBase` to merge two join if statement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14490: [ehancement](nereids) eliminate project in the post process phase
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #14490:
URL: https://github.com/apache/doris/pull/14490#issuecomment-1323525357
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.94 seconds
load time: 440 seconds
storage size: 17148813434 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221122113000_clickbench_pr_50727.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14490: [ehancement](nereids) eliminate project in the post process phase
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14490:
URL: https://github.com/apache/doris/pull/14490#issuecomment-1328290001
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14490: [ehancement](nereids) eliminate project in the post process phase
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14490:
URL: https://github.com/apache/doris/pull/14490#issuecomment-1327653304
#### `sh-checker report`
To get the full details, please check in the [job]("https://github.com/apache/doris/actions/runs/3549277497") output.
<details>
<summary>shellcheck errors</summary>
```
'shellcheck ' returned error 1 finding the following syntactical issues:
----------
In build-support/shell-check.sh line 266:
content="$(grep 'sh_checker_exclude:' "${DORIS_HOME}/.github/workflows/shellcheck.yml")"
^-----------^ SC2031 (info): DORIS_HOME was modified in a subshell. That change might be lost.
In build-support/shell-check.sh line 271:
if echo "${file:((${#DORIS_HOME} + 1))}" | grep -E "${pattern}" &>/dev/null; then
^------------^ SC2031 (info): DORIS_HOME was modified in a subshell. That change might be lost.
In build-support/shell-check.sh line 289:
pushd "${DORIS_HOME}" >/dev/null
^-----------^ SC2031 (info): DORIS_HOME was modified in a subshell. That change might be lost.
In build-support/shell-check.sh line 311:
read -r -a files < <(find_shell_scripts "${DORIS_HOME}")
^-----------^ SC2031 (info): DORIS_HOME was modified in a subshell. That change might be lost.
For more information:
https://www.shellcheck.net/wiki/SC2031 -- DORIS_HOME was modified in a subs...
----------
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
```
</details>
<details>
<summary>shfmt errors</summary>
```
'shfmt ' found no issues.
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #14490: [ehancement](nereids) eliminate project in the post process phase
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #14490:
URL: https://github.com/apache/doris/pull/14490#discussion_r1030575257
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -845,33 +851,43 @@ public PlanFragment visitPhysicalProject(PhysicalProject<? extends Plan> project
}
}
PlanFragment inputFragment = project.child(0).accept(this, context);
-
List<Expr> execExprList = project.getProjects()
.stream()
.map(e -> ExpressionTranslator.translate(e, context))
.collect(Collectors.toList());
// TODO: fix the project alias of an aliased relation.
- List<Slot> slotList = project.getOutput();
- TupleDescriptor tupleDescriptor = generateTupleDesc(slotList, null, context);
+
PlanNode inputPlanNode = inputFragment.getPlanRoot();
- // For hash join node, use vSrcToOutputSMap to describe the expression calculation, use
- // vIntermediateTupleDescList as input, and set vOutputTupleDesc as the final output.
- // TODO: HashJoinNode's be implementation is not support projection yet, remove this after when supported.
- if (inputPlanNode instanceof HashJoinNode) {
- HashJoinNode hashJoinNode = (HashJoinNode) inputPlanNode;
- hashJoinNode.setvOutputTupleDesc(tupleDescriptor);
- hashJoinNode.setvSrcToOutputSMap(execExprList);
- return inputFragment;
- }
- if (inputPlanNode instanceof NestedLoopJoinNode) {
- NestedLoopJoinNode nestedLoopJoinNode = (NestedLoopJoinNode) inputPlanNode;
- nestedLoopJoinNode.setvOutputTupleDesc(tupleDescriptor);
- nestedLoopJoinNode.setvSrcToOutputSMap(execExprList);
- return inputFragment;
+ if (isUnnecessaryProject(project)) {
+ List<NamedExpression> namedExpressions = project.getProjects();
+ for (NamedExpression n : namedExpressions) {
+ for (Expression e : n.children()) {
+ SlotRef slotRef = context.findSlotRef(((SlotReference) e).getExprId());
+ SlotReference slotReference = (SlotReference) e;
+ context.addExprIdSlotRefPair(slotReference.getExprId(), slotRef);
+ }
+ }
+ } else {
+ List<Slot> slotList = project.getOutput();
+ TupleDescriptor tupleDescriptor = generateTupleDesc(slotList, null, context);
+ inputPlanNode.setProjectList(execExprList);
+ inputPlanNode.setOutputTupleDesc(tupleDescriptor);
+ // For hash join node, use vSrcToOutputSMap to describe the expression calculation, use
+ // vIntermediateTupleDescList as input, and set vOutputTupleDesc as the final output.
+ // TODO: HashJoinNode's be implementation is not support projection yet, remove this after when supported.
+ if (inputPlanNode instanceof HashJoinNode) {
+ HashJoinNode hashJoinNode = (HashJoinNode) inputPlanNode;
+ hashJoinNode.setvOutputTupleDesc(tupleDescriptor);
+ hashJoinNode.setvSrcToOutputSMap(execExprList);
+ return inputFragment;
+ }
+ if (inputPlanNode instanceof NestedLoopJoinNode) {
+ NestedLoopJoinNode nestedLoopJoinNode = (NestedLoopJoinNode) inputPlanNode;
+ nestedLoopJoinNode.setvOutputTupleDesc(tupleDescriptor);
+ nestedLoopJoinNode.setvSrcToOutputSMap(execExprList);
+ return inputFragment;
Review Comment:
project on join node cannot be remove, this is a constraint on BE
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -128,6 +128,12 @@ public PlanFragment translatePlan(PhysicalPlan physicalPlan, PlanTranslatorConte
rootFragment = exchangeToMergeFragment(rootFragment, context);
}
List<Expr> outputExprs = Lists.newArrayList();
+ if (physicalPlan instanceof PhysicalProject) {
+ PhysicalProject project = (PhysicalProject) physicalPlan;
+ if (isUnnecessaryProject(project)) {
+ physicalPlan = (PhysicalPlan) physicalPlan.child(0);
+ }
+ }
Review Comment:
after remove the root project node, we should set output exprs's order as them in the project to ensure the result is correct.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -1162,4 +1178,19 @@ private PlanFragment constructShuffleJoin(AbstractPhysicalJoin<PhysicalPlan, Phy
return joinFragment;
}
+
+ private boolean isUnnecessaryProject(PhysicalProject project) {
+ List<NamedExpression> projects = project.getProjects();
+ for (NamedExpression p : projects) {
+ if (p.children().size() > 1) {
+ return false;
+ }
+ for (Expression e : p.children()) {
+ if (!(e instanceof SlotReference)) {
+ return false;
+ }
+ }
+ }
Review Comment:
```
1. project.output.size == project.child.output.size
2. project.output.allMatch(expr -> expr instance of SlotReference || (expr instance Alias && expr.child instance of SlotReference))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #14490: [ehancement](nereids) eliminate project in the post process phase
Posted by GitBox <gi...@apache.org>.
morrySnow merged PR #14490:
URL: https://github.com/apache/doris/pull/14490
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14490: [ehancement](nereids) eliminate project in the post process phase
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14490:
URL: https://github.com/apache/doris/pull/14490#issuecomment-1328289968
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org