You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2023/01/05 08:13:54 UTC
[GitHub] [doris] englefly opened a new pull request, #15646: [fix](nereids) bind sort key priority problem
englefly opened a new pull request, #15646:
URL: https://github.com/apache/doris/pull/15646
# Proposed changes
1. add strict mode for SlotBinder. in strict mode, binding must matches qualifier
2. for sort key binding, we first use strict mode, then loose mode
cases:
table t (int a, int b)
case1. `select a as b from t order by t.b`
case2. `select a as b from t order by b`
case3. `select a as b from t order by t.b + b`
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
4. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
5. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
6. Does it need to update dependencies:
- [ ] Yes
- [ ] No
7. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15646: [fix](nereids) bind sort key priority problem
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15646:
URL: https://github.com/apache/doris/pull/15646#issuecomment-1377281343
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15646: [fix](nereids) bind sort key priority problem
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15646:
URL: https://github.com/apache/doris/pull/15646#issuecomment-1376685765
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #15646: [fix](nereids) bind sort key priority problem
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #15646:
URL: https://github.com/apache/doris/pull/15646#discussion_r1064557552
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java:
##########
@@ -354,13 +355,32 @@ public List<Rule> buildRules() {
RuleType.BINDING_SORT_SLOT.build(
logicalSort(logicalProject()).when(Plan::canBind).thenApply(ctx -> {
LogicalSort<LogicalProject<GroupPlan>> sort = ctx.root;
+ Set<Slot> orderSlots = Sets.newHashSet();
Review Comment:
I think its ok on pattern sort(having(project)).
This pr makes binding more strict, and it is applicable to all patterns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15646: [fix](nereids) bind sort key priority problem
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #15646:
URL: https://github.com/apache/doris/pull/15646#issuecomment-1372092661
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.02 seconds
load time: 480 seconds
storage size: 17123178703 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230105112058_clickbench_pr_74202.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #15646: [fix](nereids) bind sort key priority problem
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #15646:
URL: https://github.com/apache/doris/pull/15646#discussion_r1064522228
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java:
##########
@@ -354,13 +355,32 @@ public List<Rule> buildRules() {
RuleType.BINDING_SORT_SLOT.build(
logicalSort(logicalProject()).when(Plan::canBind).thenApply(ctx -> {
LogicalSort<LogicalProject<GroupPlan>> sort = ctx.root;
+ Set<Slot> orderSlots = Sets.newHashSet();
+ sort.getOrderKeys().stream()
+ .forEach(orderKey ->
+ orderSlots.addAll(orderKey.getExpr().getInputSlots()));
+ Set<Slot> projectOutput = sort.child().getOutputSet();
+ SlotBinder strictBinderOnProject = new SlotBinder(toScope(Lists.newArrayList(projectOutput)),
+ sort, true, ctx.cascadesContext);
+ //SlotBinder looseBinderOnProject = new SlotBinder(toScope(Lists.newArrayList(projectOutput)),
+ // sort, false, ctx.cascadesContext);
+
+ Set<Slot> projectChildrenOutput = sort.child().children().stream()
+ .flatMap(plan -> plan.getOutputSet().stream())
+ .collect(Collectors.toSet());
+ SlotBinder strictBinderOnProjectChild = new SlotBinder(
+ toScope(Lists.newArrayList(projectChildrenOutput)),
+ sort, true, ctx.cascadesContext);
+ //SlotBinder looseBinderOnProjectChild = new SlotBinder(
+ // toScope(Lists.newArrayList(projectChildrenOutput)),
+ // sort, false, ctx.cascadesContext);
List<OrderKey> sortItemList = sort.getOrderKeys()
.stream()
.map(orderKey -> {
- Expression item = bind(orderKey.getExpr(), sort.children(), sort, ctx.cascadesContext);
- if (item.containsType(UnboundSlot.class)) {
- item = bind(item, sort.child().children(), sort, ctx.cascadesContext);
- }
+ Expression item = strictBinderOnProject.bind(orderKey.getExpr());
+ item = strictBinderOnProjectChild.bind(item);
+ //item = looseBinderOnProjectChild.bind(item);
+ //item = looseBinderOnProject.bind(item);
Review Comment:
done
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java:
##########
@@ -354,13 +355,32 @@ public List<Rule> buildRules() {
RuleType.BINDING_SORT_SLOT.build(
logicalSort(logicalProject()).when(Plan::canBind).thenApply(ctx -> {
LogicalSort<LogicalProject<GroupPlan>> sort = ctx.root;
+ Set<Slot> orderSlots = Sets.newHashSet();
+ sort.getOrderKeys().stream()
+ .forEach(orderKey ->
+ orderSlots.addAll(orderKey.getExpr().getInputSlots()));
+ Set<Slot> projectOutput = sort.child().getOutputSet();
+ SlotBinder strictBinderOnProject = new SlotBinder(toScope(Lists.newArrayList(projectOutput)),
+ sort, true, ctx.cascadesContext);
+ //SlotBinder looseBinderOnProject = new SlotBinder(toScope(Lists.newArrayList(projectOutput)),
+ // sort, false, ctx.cascadesContext);
+
+ Set<Slot> projectChildrenOutput = sort.child().children().stream()
+ .flatMap(plan -> plan.getOutputSet().stream())
+ .collect(Collectors.toSet());
+ SlotBinder strictBinderOnProjectChild = new SlotBinder(
+ toScope(Lists.newArrayList(projectChildrenOutput)),
+ sort, true, ctx.cascadesContext);
+ //SlotBinder looseBinderOnProjectChild = new SlotBinder(
+ // toScope(Lists.newArrayList(projectChildrenOutput)),
+ // sort, false, ctx.cascadesContext);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #15646: [fix](nereids) bind sort key priority problem
Posted by GitBox <gi...@apache.org>.
morrySnow merged PR #15646:
URL: https://github.com/apache/doris/pull/15646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #15646: [fix](nereids) bind sort key priority problem
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #15646:
URL: https://github.com/apache/doris/pull/15646#discussion_r1064145194
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java:
##########
@@ -354,13 +355,32 @@ public List<Rule> buildRules() {
RuleType.BINDING_SORT_SLOT.build(
logicalSort(logicalProject()).when(Plan::canBind).thenApply(ctx -> {
LogicalSort<LogicalProject<GroupPlan>> sort = ctx.root;
+ Set<Slot> orderSlots = Sets.newHashSet();
+ sort.getOrderKeys().stream()
+ .forEach(orderKey ->
+ orderSlots.addAll(orderKey.getExpr().getInputSlots()));
+ Set<Slot> projectOutput = sort.child().getOutputSet();
+ SlotBinder strictBinderOnProject = new SlotBinder(toScope(Lists.newArrayList(projectOutput)),
+ sort, true, ctx.cascadesContext);
+ //SlotBinder looseBinderOnProject = new SlotBinder(toScope(Lists.newArrayList(projectOutput)),
+ // sort, false, ctx.cascadesContext);
+
+ Set<Slot> projectChildrenOutput = sort.child().children().stream()
+ .flatMap(plan -> plan.getOutputSet().stream())
+ .collect(Collectors.toSet());
+ SlotBinder strictBinderOnProjectChild = new SlotBinder(
+ toScope(Lists.newArrayList(projectChildrenOutput)),
+ sort, true, ctx.cascadesContext);
+ //SlotBinder looseBinderOnProjectChild = new SlotBinder(
+ // toScope(Lists.newArrayList(projectChildrenOutput)),
+ // sort, false, ctx.cascadesContext);
Review Comment:
remove useless comments
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java:
##########
@@ -354,13 +355,32 @@ public List<Rule> buildRules() {
RuleType.BINDING_SORT_SLOT.build(
logicalSort(logicalProject()).when(Plan::canBind).thenApply(ctx -> {
LogicalSort<LogicalProject<GroupPlan>> sort = ctx.root;
+ Set<Slot> orderSlots = Sets.newHashSet();
+ sort.getOrderKeys().stream()
+ .forEach(orderKey ->
+ orderSlots.addAll(orderKey.getExpr().getInputSlots()));
+ Set<Slot> projectOutput = sort.child().getOutputSet();
+ SlotBinder strictBinderOnProject = new SlotBinder(toScope(Lists.newArrayList(projectOutput)),
+ sort, true, ctx.cascadesContext);
+ //SlotBinder looseBinderOnProject = new SlotBinder(toScope(Lists.newArrayList(projectOutput)),
+ // sort, false, ctx.cascadesContext);
Review Comment:
remove useless comments
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java:
##########
@@ -540,10 +560,17 @@ private <E extends Expression> E bind(E expr, List<Plan> inputs, Plan plan, Casc
private class SlotBinder extends SubExprAnalyzer {
private final Plan plan;
+ //strict = true: bind slot including its qualifier
+ private boolean strict;
Review Comment:
unuse var
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java:
##########
@@ -354,13 +355,32 @@ public List<Rule> buildRules() {
RuleType.BINDING_SORT_SLOT.build(
logicalSort(logicalProject()).when(Plan::canBind).thenApply(ctx -> {
LogicalSort<LogicalProject<GroupPlan>> sort = ctx.root;
+ Set<Slot> orderSlots = Sets.newHashSet();
Review Comment:
should we also update BINDING_SORT_SLOT rule on pattern `logicalSort(logicalHaving(logicalProject()))`
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java:
##########
@@ -354,13 +355,32 @@ public List<Rule> buildRules() {
RuleType.BINDING_SORT_SLOT.build(
logicalSort(logicalProject()).when(Plan::canBind).thenApply(ctx -> {
LogicalSort<LogicalProject<GroupPlan>> sort = ctx.root;
+ Set<Slot> orderSlots = Sets.newHashSet();
+ sort.getOrderKeys().stream()
+ .forEach(orderKey ->
+ orderSlots.addAll(orderKey.getExpr().getInputSlots()));
+ Set<Slot> projectOutput = sort.child().getOutputSet();
+ SlotBinder strictBinderOnProject = new SlotBinder(toScope(Lists.newArrayList(projectOutput)),
Review Comment:
remove `strict` in var's name
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java:
##########
@@ -354,13 +355,32 @@ public List<Rule> buildRules() {
RuleType.BINDING_SORT_SLOT.build(
logicalSort(logicalProject()).when(Plan::canBind).thenApply(ctx -> {
LogicalSort<LogicalProject<GroupPlan>> sort = ctx.root;
+ Set<Slot> orderSlots = Sets.newHashSet();
+ sort.getOrderKeys().stream()
+ .forEach(orderKey ->
+ orderSlots.addAll(orderKey.getExpr().getInputSlots()));
+ Set<Slot> projectOutput = sort.child().getOutputSet();
+ SlotBinder strictBinderOnProject = new SlotBinder(toScope(Lists.newArrayList(projectOutput)),
+ sort, true, ctx.cascadesContext);
+ //SlotBinder looseBinderOnProject = new SlotBinder(toScope(Lists.newArrayList(projectOutput)),
+ // sort, false, ctx.cascadesContext);
+
+ Set<Slot> projectChildrenOutput = sort.child().children().stream()
+ .flatMap(plan -> plan.getOutputSet().stream())
+ .collect(Collectors.toSet());
+ SlotBinder strictBinderOnProjectChild = new SlotBinder(
+ toScope(Lists.newArrayList(projectChildrenOutput)),
+ sort, true, ctx.cascadesContext);
+ //SlotBinder looseBinderOnProjectChild = new SlotBinder(
+ // toScope(Lists.newArrayList(projectChildrenOutput)),
+ // sort, false, ctx.cascadesContext);
List<OrderKey> sortItemList = sort.getOrderKeys()
.stream()
.map(orderKey -> {
- Expression item = bind(orderKey.getExpr(), sort.children(), sort, ctx.cascadesContext);
- if (item.containsType(UnboundSlot.class)) {
- item = bind(item, sort.child().children(), sort, ctx.cascadesContext);
- }
+ Expression item = strictBinderOnProject.bind(orderKey.getExpr());
+ item = strictBinderOnProjectChild.bind(item);
+ //item = looseBinderOnProjectChild.bind(item);
+ //item = looseBinderOnProject.bind(item);
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #15646: [fix](nereids) bind sort key priority problem
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #15646:
URL: https://github.com/apache/doris/pull/15646#discussion_r1064521948
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java:
##########
@@ -354,13 +355,32 @@ public List<Rule> buildRules() {
RuleType.BINDING_SORT_SLOT.build(
logicalSort(logicalProject()).when(Plan::canBind).thenApply(ctx -> {
LogicalSort<LogicalProject<GroupPlan>> sort = ctx.root;
+ Set<Slot> orderSlots = Sets.newHashSet();
+ sort.getOrderKeys().stream()
+ .forEach(orderKey ->
+ orderSlots.addAll(orderKey.getExpr().getInputSlots()));
+ Set<Slot> projectOutput = sort.child().getOutputSet();
+ SlotBinder strictBinderOnProject = new SlotBinder(toScope(Lists.newArrayList(projectOutput)),
+ sort, true, ctx.cascadesContext);
+ //SlotBinder looseBinderOnProject = new SlotBinder(toScope(Lists.newArrayList(projectOutput)),
+ // sort, false, ctx.cascadesContext);
Review Comment:
done
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java:
##########
@@ -540,10 +560,17 @@ private <E extends Expression> E bind(E expr, List<Plan> inputs, Plan plan, Casc
private class SlotBinder extends SubExprAnalyzer {
private final Plan plan;
+ //strict = true: bind slot including its qualifier
+ private boolean strict;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #15646: [fix](nereids) bind sort key priority problem
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #15646:
URL: https://github.com/apache/doris/pull/15646#discussion_r1064522896
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java:
##########
@@ -354,13 +355,32 @@ public List<Rule> buildRules() {
RuleType.BINDING_SORT_SLOT.build(
logicalSort(logicalProject()).when(Plan::canBind).thenApply(ctx -> {
LogicalSort<LogicalProject<GroupPlan>> sort = ctx.root;
+ Set<Slot> orderSlots = Sets.newHashSet();
+ sort.getOrderKeys().stream()
+ .forEach(orderKey ->
+ orderSlots.addAll(orderKey.getExpr().getInputSlots()));
+ Set<Slot> projectOutput = sort.child().getOutputSet();
+ SlotBinder strictBinderOnProject = new SlotBinder(toScope(Lists.newArrayList(projectOutput)),
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15646: [fix](nereids) bind sort key priority problem
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15646:
URL: https://github.com/apache/doris/pull/15646#issuecomment-1376685740
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org