You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/11/18 11:05:20 UTC
[GitHub] [doris] YangShaw opened a new pull request, #14397: [feature](nereids)support window function
YangShaw opened a new pull request, #14397:
URL: https://github.com/apache/doris/pull/14397
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1100123071
##########
fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4:
##########
@@ -332,6 +337,33 @@ functionIdentifier
| LEFT | RIGHT
;
+windowSpec
+ // todo: name for windowRef; we haven't support it
+ : name=identifier
+ | LEFT_PAREN name=identifier RIGHT_PAREN
Review Comment:
remove it, and add it back when we want to support it?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -161,7 +160,8 @@ public Plan plan(LogicalPlan plan, PhysicalProperties requireProperties, Explain
return analyzedPlan;
}
}
-
+ System.out.println(cascadesContext.getMemo().copyOut().treeString());
+ System.out.println("-------------------------------");
Review Comment:
remove println
##########
fe/fe-core/src/main/java/org/apache/doris/planner/AnalyticEvalNode.java:
##########
@@ -262,4 +260,8 @@ public String getNodeExplainString(String prefix, TExplainLevel detailLevel) {
return output.toString();
}
+
+ public void finalizeForNereids() {
+ this.substitutedPartitionExprs = partitionExprs;
+ }
Review Comment:
could we use a new constructor to avoid do finalizeForNereids?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/ExpressionTranslator.java:
##########
@@ -293,13 +294,59 @@ public Expr visitCast(Cast cast, PlanTranslatorContext context) {
@Override
public Expr visitInPredicate(InPredicate inPredicate, PlanTranslatorContext context) {
List<Expr> inList = inPredicate.getOptions().stream()
- .map(e -> translate(e, context))
+ .map(e -> e.accept(this, context))
.collect(Collectors.toList());
return new org.apache.doris.analysis.InPredicate(inPredicate.getCompareExpr().accept(this, context),
inList,
false);
}
+ @Override
+ public Expr visitWindowFunction(WindowFunction function, PlanTranslatorContext context) {
+ // translate argument types from DataType to Type
+ List<Expr> catalogArguments = function.getArguments()
+ .stream()
+ .map(arg -> arg.accept(this, context))
+ .collect(ImmutableList.toImmutableList());
+ ImmutableList<Type> argTypes = catalogArguments.stream()
+ .map(arg -> arg.getType())
+ .collect(ImmutableList.toImmutableList());
+
+ // translate argument from List<Expression> to FunctionParams
+ List<Expr> arguments = function.getArguments()
+ .stream()
+ .map(arg -> new SlotRef(arg.getDataType().toCatalogDataType(), arg.nullable()))
+ .collect(ImmutableList.toImmutableList());
+ FunctionParams windowFnParams = new FunctionParams(false, arguments);
+
+ // translate isNullable()
+ NullableMode nullableMode = function.nullable()
+ ? NullableMode.ALWAYS_NULLABLE
+ : NullableMode.ALWAYS_NOT_NULLABLE;
+
+ // translate function from WindowFunction to old AggregateFunction
+ boolean isAnalyticFunction = true;
+ org.apache.doris.catalog.AggregateFunction catalogFunction = new org.apache.doris.catalog.AggregateFunction(
+ new FunctionName(function.getName()), argTypes,
+ function.getDataType().toCatalogDataType(),
+ function.getDataType().toCatalogDataType(),
+ function.hasVarArguments(),
+ null, "", "", null, "",
+ null, "", null, false,
+ isAnalyticFunction, false, TFunctionBinaryType.BUILTIN,
+ true, true, nullableMode
+ );
+
+ // generate FunctionCallExpr
+ boolean isMergeFn = false;
+ FunctionCallExpr functionCallExpr =
+ new FunctionCallExpr(catalogFunction, windowFnParams, windowFnParams, isMergeFn, catalogArguments);
+ functionCallExpr.setIsAnalyticFnCall(true);
+ return functionCallExpr;
+
+ }
+
+ // TODO: Supports for `distinct`
Review Comment:
what's this TODO mean?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/WindowFrame.java:
##########
@@ -0,0 +1,280 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.shape.LeafExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * window frame
+ */
+public class WindowFrame extends Expression implements PropagateNullable, LeafExpression {
+
+ private FrameUnitsType frameUnits;
+
+ private FrameBoundary leftBoundary;
+
+ private FrameBoundary rightBoundary;
+
+ public WindowFrame(FrameUnitsType frameUnits, FrameBoundary leftBoundary) {
+ this(frameUnits, leftBoundary, new FrameBoundary(FrameBoundType.EMPTY_BOUNDARY));
+ }
+
+ public WindowFrame(FrameUnitsType frameUnits, FrameBoundary leftBoundary, FrameBoundary rightBoundary) {
+ this.frameUnits = frameUnits;
+ this.leftBoundary = leftBoundary;
+ this.rightBoundary = rightBoundary;
+ }
+
+ public FrameUnitsType getFrameUnits() {
+ return frameUnits;
+ }
+
+ public FrameBoundary getLeftBoundary() {
+ return leftBoundary;
+ }
+
+ public FrameBoundary getRightBoundary() {
+ return rightBoundary;
+ }
+
+ public void setRightBoundary(FrameBoundary rightBoundary) {
+ this.rightBoundary = rightBoundary;
+ }
+
+ /**
+ * reverse left & right boundary; reverse each boundary's upper and lower bound
+ */
+ public WindowFrame reverseWindow() {
+ return new WindowFrame(frameUnits, rightBoundary.reverse(), leftBoundary.reverse());
+ }
+
+ public WindowFrame withRightBoundary(FrameBoundary newRightBoundary) {
+ return new WindowFrame(frameUnits, leftBoundary, newRightBoundary);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ WindowFrame other = (WindowFrame) o;
+ return Objects.equals(this.frameUnits, other.frameUnits)
+ && Objects.equals(this.leftBoundary, other.leftBoundary)
+ && Objects.equals(this.rightBoundary, other.rightBoundary);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(frameUnits, leftBoundary, rightBoundary);
+ }
+
+ @Override
+ public String toSql() {
+ StringBuilder sb = new StringBuilder();
+ sb.append(frameUnits + " ");
+ if (rightBoundary != null) {
+ sb.append("BETWEEN " + leftBoundary + " AND " + rightBoundary);
+ } else {
+ sb.append(leftBoundary);
+ }
+ return sb.toString();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ sb.append("WindowFrame(");
+ sb.append(frameUnits + ", ");
+ sb.append(leftBoundary + ", ");
+ if (rightBoundary != null) {
+ sb.append(rightBoundary);
+ }
+ sb.append(")");
+ return sb.toString();
+ }
+
+ @Override
+ public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
+ return visitor.visitWindowFrame(this, context);
+ }
+
+ /**
+ * frame units types
+ */
+ public enum FrameUnitsType {
+ ROWS(),
+ RANGE()
+ }
+
+ /**
+ * frame boundary
+ */
+ public static class FrameBoundary {
+
+ private Optional<Expression> boundOffset;
+ private FrameBoundType frameBoundType;
Review Comment:
final
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/WindowFrame.java:
##########
@@ -0,0 +1,280 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.shape.LeafExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * window frame
+ */
+public class WindowFrame extends Expression implements PropagateNullable, LeafExpression {
+
+ private FrameUnitsType frameUnits;
+
+ private FrameBoundary leftBoundary;
+
+ private FrameBoundary rightBoundary;
+
+ public WindowFrame(FrameUnitsType frameUnits, FrameBoundary leftBoundary) {
+ this(frameUnits, leftBoundary, new FrameBoundary(FrameBoundType.EMPTY_BOUNDARY));
+ }
+
+ public WindowFrame(FrameUnitsType frameUnits, FrameBoundary leftBoundary, FrameBoundary rightBoundary) {
+ this.frameUnits = frameUnits;
+ this.leftBoundary = leftBoundary;
+ this.rightBoundary = rightBoundary;
+ }
+
+ public FrameUnitsType getFrameUnits() {
+ return frameUnits;
+ }
+
+ public FrameBoundary getLeftBoundary() {
+ return leftBoundary;
+ }
+
+ public FrameBoundary getRightBoundary() {
+ return rightBoundary;
+ }
+
+ public void setRightBoundary(FrameBoundary rightBoundary) {
+ this.rightBoundary = rightBoundary;
+ }
+
+ /**
+ * reverse left & right boundary; reverse each boundary's upper and lower bound
+ */
+ public WindowFrame reverseWindow() {
+ return new WindowFrame(frameUnits, rightBoundary.reverse(), leftBoundary.reverse());
+ }
+
+ public WindowFrame withRightBoundary(FrameBoundary newRightBoundary) {
+ return new WindowFrame(frameUnits, leftBoundary, newRightBoundary);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ WindowFrame other = (WindowFrame) o;
+ return Objects.equals(this.frameUnits, other.frameUnits)
+ && Objects.equals(this.leftBoundary, other.leftBoundary)
+ && Objects.equals(this.rightBoundary, other.rightBoundary);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(frameUnits, leftBoundary, rightBoundary);
+ }
+
+ @Override
+ public String toSql() {
+ StringBuilder sb = new StringBuilder();
+ sb.append(frameUnits + " ");
+ if (rightBoundary != null) {
+ sb.append("BETWEEN " + leftBoundary + " AND " + rightBoundary);
+ } else {
+ sb.append(leftBoundary);
+ }
+ return sb.toString();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ sb.append("WindowFrame(");
+ sb.append(frameUnits + ", ");
+ sb.append(leftBoundary + ", ");
+ if (rightBoundary != null) {
+ sb.append(rightBoundary);
+ }
+ sb.append(")");
+ return sb.toString();
+ }
+
+ @Override
+ public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
+ return visitor.visitWindowFrame(this, context);
+ }
+
+ /**
+ * frame units types
+ */
+ public enum FrameUnitsType {
+ ROWS(),
+ RANGE()
Review Comment:
```suggestion
ROWS,
RANGE
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/WindowFrame.java:
##########
@@ -0,0 +1,280 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.shape.LeafExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * window frame
+ */
+public class WindowFrame extends Expression implements PropagateNullable, LeafExpression {
+
+ private FrameUnitsType frameUnits;
+
+ private FrameBoundary leftBoundary;
+
+ private FrameBoundary rightBoundary;
+
+ public WindowFrame(FrameUnitsType frameUnits, FrameBoundary leftBoundary) {
+ this(frameUnits, leftBoundary, new FrameBoundary(FrameBoundType.EMPTY_BOUNDARY));
+ }
+
+ public WindowFrame(FrameUnitsType frameUnits, FrameBoundary leftBoundary, FrameBoundary rightBoundary) {
+ this.frameUnits = frameUnits;
+ this.leftBoundary = leftBoundary;
+ this.rightBoundary = rightBoundary;
+ }
+
+ public FrameUnitsType getFrameUnits() {
+ return frameUnits;
+ }
+
+ public FrameBoundary getLeftBoundary() {
+ return leftBoundary;
+ }
+
+ public FrameBoundary getRightBoundary() {
+ return rightBoundary;
+ }
+
+ public void setRightBoundary(FrameBoundary rightBoundary) {
+ this.rightBoundary = rightBoundary;
+ }
+
+ /**
+ * reverse left & right boundary; reverse each boundary's upper and lower bound
+ */
+ public WindowFrame reverseWindow() {
+ return new WindowFrame(frameUnits, rightBoundary.reverse(), leftBoundary.reverse());
+ }
+
+ public WindowFrame withRightBoundary(FrameBoundary newRightBoundary) {
+ return new WindowFrame(frameUnits, leftBoundary, newRightBoundary);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ WindowFrame other = (WindowFrame) o;
+ return Objects.equals(this.frameUnits, other.frameUnits)
+ && Objects.equals(this.leftBoundary, other.leftBoundary)
+ && Objects.equals(this.rightBoundary, other.rightBoundary);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(frameUnits, leftBoundary, rightBoundary);
+ }
+
+ @Override
+ public String toSql() {
+ StringBuilder sb = new StringBuilder();
+ sb.append(frameUnits + " ");
+ if (rightBoundary != null) {
+ sb.append("BETWEEN " + leftBoundary + " AND " + rightBoundary);
+ } else {
+ sb.append(leftBoundary);
+ }
+ return sb.toString();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ sb.append("WindowFrame(");
+ sb.append(frameUnits + ", ");
+ sb.append(leftBoundary + ", ");
+ if (rightBoundary != null) {
+ sb.append(rightBoundary);
+ }
+ sb.append(")");
+ return sb.toString();
+ }
+
+ @Override
+ public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
+ return visitor.visitWindowFrame(this, context);
+ }
+
+ /**
+ * frame units types
+ */
+ public enum FrameUnitsType {
+ ROWS(),
+ RANGE()
+ }
+
+ /**
+ * frame boundary
+ */
+ public static class FrameBoundary {
+
+ private Optional<Expression> boundOffset;
+ private FrameBoundType frameBoundType;
+
+ public FrameBoundary(FrameBoundType frameBoundType) {
+ this.boundOffset = Optional.empty();
+ this.frameBoundType = frameBoundType;
+ }
+
+ public FrameBoundary(Optional<Expression> boundOffset, FrameBoundType frameBoundType) {
+ this.boundOffset = boundOffset;
+ this.frameBoundType = frameBoundType;
+ }
+
+ public static FrameBoundary newPrecedingBoundary() {
+ return new FrameBoundary(FrameBoundType.UNBOUNDED_PRECEDING);
+ }
+
+ public static FrameBoundary newPrecedingBoundary(Expression boundValue) {
+ return new FrameBoundary(Optional.of(boundValue), FrameBoundType.PRECEDING);
+ }
+
+ public static FrameBoundary newFollowingBoundary() {
+ return new FrameBoundary(FrameBoundType.UNBOUNDED_FOLLOWING);
+ }
+
+ public static FrameBoundary newFollowingBoundary(Expression boundValue) {
+ return new FrameBoundary(Optional.of(boundValue), FrameBoundType.FOLLOWING);
+ }
+
+ public static FrameBoundary newCurrentRowBoundary() {
+ return new FrameBoundary(FrameBoundType.CURRENT_ROW);
+ }
+
+ public boolean is(FrameBoundType otherType) {
+ return this.frameBoundType == otherType;
+ }
+
+ public boolean isNot(FrameBoundType otherType) {
+ return this.frameBoundType != otherType;
+ }
+
+ public boolean isNull() {
+ return this.frameBoundType == FrameBoundType.EMPTY_BOUNDARY;
+ }
+
+ public boolean hasOffset() {
+ return frameBoundType == FrameBoundType.PRECEDING || frameBoundType == FrameBoundType.FOLLOWING;
+ }
+
+ public boolean asPreceding() {
+ return frameBoundType == FrameBoundType.PRECEDING || frameBoundType == FrameBoundType.UNBOUNDED_PRECEDING;
+ }
+
+ public boolean asFollowing() {
+ return frameBoundType == FrameBoundType.FOLLOWING || frameBoundType == FrameBoundType.UNBOUNDED_FOLLOWING;
+ }
+
+ public FrameBoundary reverse() {
+ return new FrameBoundary(boundOffset, frameBoundType.reverse());
+ }
+
+ public FrameBoundType getFrameBoundType() {
+ return frameBoundType;
+ }
+
+ public Optional<Expression> getBoundOffset() {
+ return boundOffset;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ boundOffset.ifPresent(value -> sb.append(value + " "));
+ sb.append(frameBoundType);
+
+ return sb.toString();
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ FrameBoundary other = (FrameBoundary) o;
+ return Objects.equals(this.frameBoundType, other.frameBoundType)
+ && Objects.equals(this.boundOffset, other.boundOffset);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(boundOffset, frameBoundType);
+ }
+ }
+
+ /**
+ * frame bound types
+ */
+ public enum FrameBoundType {
+
+ UNBOUNDED_PRECEDING("UNBOUNDED_PRECEDING"),
+ UNBOUNDED_FOLLOWING("UNBOUNDED_FOLLOWING"),
+ CURRENT_ROW("CURRENT_ROW"),
+ PRECEDING("PRECEDING"),
+ FOLLOWING("FOLLOWING"),
+
+ // represents that the boundary is null. We use this value as default
+ // to avoid checking if a boundary is null frequently.
+ EMPTY_BOUNDARY("EMPTY_BOUNDARY");
+
+ private final String description;
Review Comment:
description is useless
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,545 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractAndNormalizeWindowExpression;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.OrderExpression;
+import org.apache.doris.nereids.trees.expressions.WindowExpression;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractAndNormalizeWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isChecked).then(this::implement)
Review Comment:
why need isChecked?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/WindowFrame.java:
##########
@@ -0,0 +1,280 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.shape.LeafExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * window frame
+ */
+public class WindowFrame extends Expression implements PropagateNullable, LeafExpression {
+
+ private FrameUnitsType frameUnits;
+
+ private FrameBoundary leftBoundary;
+
+ private FrameBoundary rightBoundary;
Review Comment:
final
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/NereidsRewriteJobExecutor.java:
##########
@@ -90,6 +92,11 @@ public NereidsRewriteJobExecutor(CascadesContext cascadesContext) {
.add(bottomUpBatch(ImmutableList.of(new AdjustAggregateNullableForEmptySet())))
.add(topDownBatch(ImmutableList.of(new EliminateGroupByConstant())))
.add(topDownBatch(ImmutableList.of(new NormalizeAggregate())))
+ .add(topDownBatch(ImmutableList.of(new ExtractAndNormalizeWindowExpression())))
+ //execute NormalizeAggregate() again to resolve nested AggregateFunctions in WindowExpression,
+ //e.g. sum(sum(c1)) over(partition by avg(c1))
+ .add(topDownBatch(ImmutableList.of(new NormalizeAggregate())))
Review Comment:
it is weird, i think after normalize aggregate we should get
```
project(sum(a1) over (partition by a2))
+---agg(sum(c1) as a1, avg(c1) as c2)
```
and then we do extract window expression
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] YangShaw commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "YangShaw (via GitHub)" <gi...@apache.org>.
YangShaw commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1099844627
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
+ // todo: partitionByEq and orderByEq?
+ // requiredProperties:
+ // Distribution: partitionKeys
+ // Order: requiredOrderKeys
+ PhysicalWindow physicalWindow = new PhysicalWindow<>(
+ logicalWindow.getOutputExpressions(),
+ logicalWindow.getWindowExpressions(),
+ windowFrameGroup,
+ logicalWindow.getLogicalProperties(),
+ root);
+ // todo: add isAnalyticSort to physicalWindow
+ if (windowFrameGroup.partitionKeyList.isEmpty() && requiredOrderKeys.isEmpty()) {
+ return physicalWindow.withRequirePropertiesAndChild(RequireProperties.followParent(), root);
+ }
+
+ PhysicalProperties properties;
+ if (windowFrameGroup.partitionKeyList.isEmpty()) {
+ properties = new PhysicalProperties(new OrderSpec(requiredOrderKeys));
+ } else {
+ // todo: add new ShuffleType for window, like ShuffleType.WINDOW
+ properties = PhysicalProperties.createHash(
+ windowFrameGroup.partitionKeyList, DistributionSpecHash.ShuffleType.ENFORCED);
+ // requiredOrderKeys contain partitionKeys, so there is no need to check if requiredOrderKeys.isEmpty()
+ properties = properties.withOrderSpec(new OrderSpec(requiredOrderKeys));
+ }
+
+ RequireProperties requireProperties = RequireProperties.of(properties);
+ return physicalWindow.withRequirePropertiesAndChild(requireProperties, root);
+ }
+
+ /* ********************************************************************************************
+ * WindowFunctionRelatedGroups
+ * ******************************************************************************************** */
+
+ // todo: can we simplify the following three algorithms?
+ private List<WindowFrameGroup> createCommonWindowFrameGroups(List<NamedExpression> windowList) {
+ List<WindowFrameGroup> windowFrameGroupList = Lists.newArrayList();
+ for (int i = 0; i < windowList.size(); i++) {
+ NamedExpression windowAlias = windowList.get(i);
+
+ boolean matched = false;
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ if (windowFrameGroup.isCompatible(windowAlias)) {
+ windowFrameGroup.addGroupMember(windowAlias);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ windowFrameGroupList.add(new WindowFrameGroup(windowAlias));
+ }
+ }
+
+ for (WindowFrameGroup wfg : windowFrameGroupList) {
+ wfg.setTupleSize(wfg.groupList.stream().mapToInt(window -> window.child(0).getDataType().width()).sum());
+ }
+
+ return windowFrameGroupList;
+ }
+
+ private List<OrderKeyGroup> createCommonOrderKeyGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ List<OrderKeyGroup> orderKeyGroupList = Lists.newArrayList();
+
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ boolean matched = false;
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ if (orderKeyGroup.isCompatible(windowFrameGroup)) {
+ orderKeyGroup.addGroupMember(windowFrameGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ orderKeyGroupList.add(new OrderKeyGroup(windowFrameGroup));
+ }
+ }
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ okg.setTupleSize(okg.getGroupList().stream().mapToInt(WindowFrameGroup::getTupleSize).sum());
+ }
+
+ return orderKeyGroupList;
+ }
+
+ private List<PartitionKeyGroup> createCommonPartitionKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ List<PartitionKeyGroup> partitionKeyGroupList = Lists.newArrayList();
+
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ boolean matched = false;
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ if (partitionKeyGroup.isCompatible(orderKeyGroup)) {
+ partitionKeyGroup.addGroupMember(orderKeyGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ partitionKeyGroupList.add(new PartitionKeyGroup(orderKeyGroup));
+ }
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ pkg.setTupleSize(pkg.getGroupList().stream().mapToInt(OrderKeyGroup::getTupleSize).sum());
+ }
+
+ return partitionKeyGroupList;
+ }
+
+ private void mergeOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (OrderKeyGroup okg1 : orderKeyGroupList) {
+ for (OrderKeyGroup okg2 : orderKeyGroupList) {
+ if (okg1 != okg2 && okg2.isPrefixOf(okg1)) {
+ // okg2 ∈ okg1
+ okg1.absorb(okg2);
+ orderKeyGroupList.remove(okg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ private void mergePartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (PartitionKeyGroup pkg1 : partitionKeyGroupList) {
+ for (PartitionKeyGroup pkg2 : partitionKeyGroupList) {
+ if (pkg1 != pkg2) {
+ // todo:
Review Comment:
I have completed this todo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #14397: [feature](nereids)support window function
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #14397:
URL: https://github.com/apache/doris/pull/14397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] YangShaw commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "YangShaw (via GitHub)" <gi...@apache.org>.
YangShaw commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1094134457
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalWindow.java:
##########
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.Window;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * logical node to deal with window functions
+ */
+public class LogicalWindow<CHILD_TYPE extends Plan> extends LogicalUnary<CHILD_TYPE> implements Window {
+
+ private List<NamedExpression> outputExpressions;
Review Comment:
have removed windowExpressions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] YangShaw commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "YangShaw (via GitHub)" <gi...@apache.org>.
YangShaw commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1095317560
##########
fe/fe-core/src/test/java/org/apache/doris/nereids/parser/NereidsParserTest.java:
##########
@@ -106,6 +106,16 @@ public void testParseCTE() {
Assertions.assertEquals(columnAliases.get().size(), 2);
}
+ @Test
+ public void testParseWindowFunctions() {
+ NereidsParser nereidsParser = new NereidsParser();
+ LogicalPlan logicalPlan;
+
+ String windowSql1 = "select rank() over(partition by k1 order by k1) as ranking from t1";
+ logicalPlan = nereidsParser.parseSingle(windowSql1);
+ System.out.println(logicalPlan.treeString());
Review Comment:
yes, i will complete it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] YangShaw commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "YangShaw (via GitHub)" <gi...@apache.org>.
YangShaw commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1099843860
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
+ // todo: partitionByEq and orderByEq?
+ // requiredProperties:
+ // Distribution: partitionKeys
+ // Order: requiredOrderKeys
+ PhysicalWindow physicalWindow = new PhysicalWindow<>(
+ logicalWindow.getOutputExpressions(),
+ logicalWindow.getWindowExpressions(),
+ windowFrameGroup,
+ logicalWindow.getLogicalProperties(),
+ root);
+ // todo: add isAnalyticSort to physicalWindow
+ if (windowFrameGroup.partitionKeyList.isEmpty() && requiredOrderKeys.isEmpty()) {
+ return physicalWindow.withRequirePropertiesAndChild(RequireProperties.followParent(), root);
+ }
+
+ PhysicalProperties properties;
+ if (windowFrameGroup.partitionKeyList.isEmpty()) {
+ properties = new PhysicalProperties(new OrderSpec(requiredOrderKeys));
+ } else {
+ // todo: add new ShuffleType for window, like ShuffleType.WINDOW
+ properties = PhysicalProperties.createHash(
+ windowFrameGroup.partitionKeyList, DistributionSpecHash.ShuffleType.ENFORCED);
+ // requiredOrderKeys contain partitionKeys, so there is no need to check if requiredOrderKeys.isEmpty()
+ properties = properties.withOrderSpec(new OrderSpec(requiredOrderKeys));
+ }
+
+ RequireProperties requireProperties = RequireProperties.of(properties);
+ return physicalWindow.withRequirePropertiesAndChild(requireProperties, root);
+ }
+
+ /* ********************************************************************************************
+ * WindowFunctionRelatedGroups
+ * ******************************************************************************************** */
+
+ // todo: can we simplify the following three algorithms?
+ private List<WindowFrameGroup> createCommonWindowFrameGroups(List<NamedExpression> windowList) {
+ List<WindowFrameGroup> windowFrameGroupList = Lists.newArrayList();
+ for (int i = 0; i < windowList.size(); i++) {
+ NamedExpression windowAlias = windowList.get(i);
+
+ boolean matched = false;
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ if (windowFrameGroup.isCompatible(windowAlias)) {
+ windowFrameGroup.addGroupMember(windowAlias);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ windowFrameGroupList.add(new WindowFrameGroup(windowAlias));
+ }
+ }
+
+ for (WindowFrameGroup wfg : windowFrameGroupList) {
+ wfg.setTupleSize(wfg.groupList.stream().mapToInt(window -> window.child(0).getDataType().width()).sum());
+ }
+
+ return windowFrameGroupList;
+ }
+
+ private List<OrderKeyGroup> createCommonOrderKeyGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ List<OrderKeyGroup> orderKeyGroupList = Lists.newArrayList();
+
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ boolean matched = false;
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ if (orderKeyGroup.isCompatible(windowFrameGroup)) {
+ orderKeyGroup.addGroupMember(windowFrameGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ orderKeyGroupList.add(new OrderKeyGroup(windowFrameGroup));
+ }
+ }
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ okg.setTupleSize(okg.getGroupList().stream().mapToInt(WindowFrameGroup::getTupleSize).sum());
+ }
+
+ return orderKeyGroupList;
+ }
+
+ private List<PartitionKeyGroup> createCommonPartitionKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ List<PartitionKeyGroup> partitionKeyGroupList = Lists.newArrayList();
+
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ boolean matched = false;
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ if (partitionKeyGroup.isCompatible(orderKeyGroup)) {
+ partitionKeyGroup.addGroupMember(orderKeyGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ partitionKeyGroupList.add(new PartitionKeyGroup(orderKeyGroup));
+ }
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ pkg.setTupleSize(pkg.getGroupList().stream().mapToInt(OrderKeyGroup::getTupleSize).sum());
+ }
+
+ return partitionKeyGroupList;
+ }
+
+ private void mergeOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (OrderKeyGroup okg1 : orderKeyGroupList) {
+ for (OrderKeyGroup okg2 : orderKeyGroupList) {
+ if (okg1 != okg2 && okg2.isPrefixOf(okg1)) {
+ // okg2 ∈ okg1
+ okg1.absorb(okg2);
+ orderKeyGroupList.remove(okg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
Review Comment:
yes, I have fixed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14397: [feature](nereids)support window function
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14397:
URL: https://github.com/apache/doris/pull/14397#issuecomment-1427905270
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1091595083
##########
fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4:
##########
@@ -327,6 +332,32 @@ functionIdentifier
| LEFT | RIGHT
;
+windowSpec
+ : name=identifier
+ | LEFT_PAREN name=identifier RIGHT_PAREN
+ | LEFT_PAREN
+ partitionClause?
+ sortClause?
+ windowFrame?
+ RIGHT_PAREN
Review Comment:
does this mean the empty parentheses is legal?
```
SELECT ROW_NUMBER() OVER () FROM t;
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -182,12 +181,13 @@ public Plan plan(LogicalPlan plan, PhysicalProperties requireProperties, Explain
PhysicalPlan physicalPlan = chooseBestPlan(getRoot(), requireProperties);
- // post-process physical plan out of memo, just for future use.
physicalPlan = postProcess(physicalPlan);
if (explainLevel == ExplainLevel.OPTIMIZED_PLAN || explainLevel == ExplainLevel.ALL_PLAN) {
optimizedPlan = physicalPlan;
}
+ System.out.println(physicalPlan.treeString());
+
Review Comment:
remove it~
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Window.java:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.DataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * represents window function. WindowFunction of this window is saved as Window's child,
+ * which is an UnboundFunction at first and will be analyzed as relevant BoundFunction
+ * (can be a WindowFunction or AggregateFunction) after BindFunction.
+ */
+public class Window extends Expression implements UnaryExpression, PropagateNullable {
+
+ private Optional<List<Expression>> partitionKeyList;
+
+ private Optional<List<OrderKey>> orderKeyList;
+
+ private Optional<WindowFrame> windowFrame;
Review Comment:
final
##########
fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4:
##########
@@ -327,6 +332,32 @@ functionIdentifier
| LEFT | RIGHT
;
+windowSpec
+ : name=identifier
+ | LEFT_PAREN name=identifier RIGHT_PAREN
+ | LEFT_PAREN
+ partitionClause?
+ sortClause?
+ windowFrame?
+ RIGHT_PAREN
+ ;
+
+windowFrame
+ : frameUnits start=frameBound
+ | frameUnits BETWEEN start=frameBound AND end=frameBound
+ ;
+
+frameUnits
+ : ROWS
+ | RANGE
+ ;
+
+frameBound
Review Comment:
```suggestion
frameBoundary
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Window.java:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.DataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * represents window function. WindowFunction of this window is saved as Window's child,
+ * which is an UnboundFunction at first and will be analyzed as relevant BoundFunction
+ * (can be a WindowFunction or AggregateFunction) after BindFunction.
+ */
+public class Window extends Expression implements UnaryExpression, PropagateNullable {
Review Comment:
i think we should treat all window spec as window expression's children. then we could do expression rewrite easily
refer to spark~
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -945,15 +954,99 @@ public Expression visitFunctionCall(DorisParser.FunctionCallContext ctx) {
if (!unboundStars.get(0).getQualifier().isEmpty()) {
throw new ParseException("'*' can not has qualifier: " + unboundStars.size(), ctx);
}
+ if (ctx.windowSpec() != null) {
+ throw new ParseException(
+ "COUNT(*) isn't supported as window function; can use COUNT(col)", ctx);
+ }
return new Count();
}
throw new ParseException("'*' can only be used in conjunction with COUNT: " + functionName, ctx);
} else {
- return new UnboundFunction(functionName, isDistinct, params);
+ UnboundFunction function = new UnboundFunction(functionName, isDistinct, params);
+ if (ctx.windowSpec() != null) {
+ if (isDistinct) {
+ throw new ParseException("DISTINCT not allowed in window function: " + functionName, ctx);
Review Comment:
throw error same as legacy planner
```
DISTINCT not allowed in analytic function: count(DISTINCT 1)
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Window.java:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.DataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * represents window function. WindowFunction of this window is saved as Window's child,
+ * which is an UnboundFunction at first and will be analyzed as relevant BoundFunction
+ * (can be a WindowFunction or AggregateFunction) after BindFunction.
+ */
+public class Window extends Expression implements UnaryExpression, PropagateNullable {
+
+ private Optional<List<Expression>> partitionKeyList;
Review Comment:
```suggestion
private final Optional<List<Expression>> partitionKeys;
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Window.java:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.DataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * represents window function. WindowFunction of this window is saved as Window's child,
+ * which is an UnboundFunction at first and will be analyzed as relevant BoundFunction
+ * (can be a WindowFunction or AggregateFunction) after BindFunction.
+ */
+public class Window extends Expression implements UnaryExpression, PropagateNullable {
Review Comment:
rename to `WindowExpression` to distinguished from window plan~
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Window.java:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.DataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * represents window function. WindowFunction of this window is saved as Window's child,
+ * which is an UnboundFunction at first and will be analyzed as relevant BoundFunction
+ * (can be a WindowFunction or AggregateFunction) after BindFunction.
+ */
+public class Window extends Expression implements UnaryExpression, PropagateNullable {
+
+ private Optional<List<Expression>> partitionKeyList;
+
+ private Optional<List<OrderKey>> orderKeyList;
Review Comment:
```suggestion
private final Optional<List<OrderKey>> orderKeys;
```
##########
fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4:
##########
@@ -327,6 +332,32 @@ functionIdentifier
| LEFT | RIGHT
;
+windowSpec
+ : name=identifier
+ | LEFT_PAREN name=identifier RIGHT_PAREN
Review Comment:
does legacy planner support windowRef?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -945,15 +954,99 @@ public Expression visitFunctionCall(DorisParser.FunctionCallContext ctx) {
if (!unboundStars.get(0).getQualifier().isEmpty()) {
throw new ParseException("'*' can not has qualifier: " + unboundStars.size(), ctx);
}
+ if (ctx.windowSpec() != null) {
+ throw new ParseException(
+ "COUNT(*) isn't supported as window function; can use COUNT(col)", ctx);
+ }
Review Comment:
add a todo, after we remove legacy planner, we should support count(*) as window function~
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalWindow.java:
##########
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.Window;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * logical node to deal with window functions
+ */
+public class LogicalWindow<CHILD_TYPE extends Plan> extends LogicalUnary<CHILD_TYPE> implements Window {
+
+ private List<NamedExpression> outputExpressions;
+
+ private List<NamedExpression> windowExpressions;
+
+ private boolean isNormalized;
+
+ private boolean isChecked;
Review Comment:
final
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalWindow.java:
##########
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.Window;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * logical node to deal with window functions
+ */
+public class LogicalWindow<CHILD_TYPE extends Plan> extends LogicalUnary<CHILD_TYPE> implements Window {
+
+ private List<NamedExpression> outputExpressions;
+
+ private List<NamedExpression> windowExpressions;
+
+ private boolean isNormalized;
+
+ private boolean isChecked;
+
+ /**
+ * used in step Analyze
+ */
+ public LogicalWindow(List<NamedExpression> outputExpressions, CHILD_TYPE child) {
+ this(outputExpressions, null, false, false, Optional.empty(), Optional.empty(), child);
+ }
+
+ public LogicalWindow(List<NamedExpression> outputExpressions, List<NamedExpression> windowExpressions,
+ boolean isNormalized, boolean isChecked, CHILD_TYPE child) {
+ this(outputExpressions, windowExpressions, isNormalized, isChecked, Optional.empty(),
+ Optional.empty(), child);
+ }
+
+ public LogicalWindow(List<NamedExpression> outputExpressions, List<NamedExpression> windowExpressions,
+ boolean isNormalized, boolean isChecked,
+ Optional<GroupExpression> groupExpression, Optional<LogicalProperties> logicalProperties,
+ CHILD_TYPE child) {
+ super(PlanType.LOGICAL_WINDOW, groupExpression, logicalProperties, child);
+ this.outputExpressions = outputExpressions;
+ this.windowExpressions = windowExpressions;
Review Comment:
ImmutableList.copyOf(Objects.requireNotNull(..., "...."))
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalWindow.java:
##########
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.Window;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * logical node to deal with window functions
+ */
+public class LogicalWindow<CHILD_TYPE extends Plan> extends LogicalUnary<CHILD_TYPE> implements Window {
+
+ private List<NamedExpression> outputExpressions;
Review Comment:
why we need a explicit `outputExpressions`? i think output could get from `windowExpressions` by `windowExpressions.stream().map(e -> e.toSlot()).collect(Collectors.toList());`
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalWindow.java:
##########
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.Window;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * logical node to deal with window functions
+ */
+public class LogicalWindow<CHILD_TYPE extends Plan> extends LogicalUnary<CHILD_TYPE> implements Window {
+
+ private List<NamedExpression> outputExpressions;
+
+ private List<NamedExpression> windowExpressions;
+
+ private boolean isNormalized;
+
+ private boolean isChecked;
+
+ /**
+ * used in step Analyze
+ */
+ public LogicalWindow(List<NamedExpression> outputExpressions, CHILD_TYPE child) {
+ this(outputExpressions, null, false, false, Optional.empty(), Optional.empty(), child);
+ }
+
+ public LogicalWindow(List<NamedExpression> outputExpressions, List<NamedExpression> windowExpressions,
+ boolean isNormalized, boolean isChecked, CHILD_TYPE child) {
+ this(outputExpressions, windowExpressions, isNormalized, isChecked, Optional.empty(),
+ Optional.empty(), child);
+ }
+
+ public LogicalWindow(List<NamedExpression> outputExpressions, List<NamedExpression> windowExpressions,
+ boolean isNormalized, boolean isChecked,
+ Optional<GroupExpression> groupExpression, Optional<LogicalProperties> logicalProperties,
+ CHILD_TYPE child) {
+ super(PlanType.LOGICAL_WINDOW, groupExpression, logicalProperties, child);
+ this.outputExpressions = outputExpressions;
+ this.windowExpressions = windowExpressions;
+ this.isNormalized = isNormalized;
+ this.isChecked = isChecked;
+ }
+
+ public boolean isNormalized() {
+ return isNormalized;
+ }
+
+ public boolean isChecked() {
+ return isChecked;
+ }
+
+ @Override
+ public List<NamedExpression> getOutputExpressions() {
+ return outputExpressions;
+ }
+
+ public List<NamedExpression> getWindowExpressions() {
+ return windowExpressions;
+ }
+
+ /**
+ * get all expressions used in LogicalWindow
+ */
+ @Override
+ public List<? extends Expression> getExpressions() {
+ ImmutableList.Builder<Expression> builder = new ImmutableList.Builder<>();
+ builder.addAll(outputExpressions);
+ if (windowExpressions != null) {
Review Comment:
why windowExpressions could be null?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/algebra/Window.java:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.algebra;
+
+import org.apache.doris.analysis.AnalyticWindow;
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.translator.ExpressionTranslator;
+import org.apache.doris.nereids.glue.translator.PlanTranslatorContext;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.expressions.functions.window.FrameBoundType;
+import org.apache.doris.nereids.trees.expressions.functions.window.FrameBoundary;
+import org.apache.doris.nereids.trees.expressions.functions.window.FrameUnitsType;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * interface for LogicalWindow and PhysicalWindow
+ */
+public interface Window {
+
+ List<NamedExpression> getOutputExpressions();
+
+ default List<Expression> extractExpressionsFromWindow(List<NamedExpression> windowExpressions) {
+ return windowExpressions.stream().map(expression -> expression.child(0))
+ .map(org.apache.doris.nereids.trees.expressions.Window.class::cast)
+ .flatMap(window -> window.getExpressionsInWindowSpec().stream())
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * translate WindowFrame to AnalyticWindow
+ */
+ default AnalyticWindow translateWindowFrame(WindowFrame windowFrame, PlanTranslatorContext context) {
+ FrameUnitsType frameUnits = windowFrame.getFrameUnits();
+ FrameBoundary leftBoundary = windowFrame.getLeftBoundary();
+ FrameBoundary rightBoundary = windowFrame.getRightBoundary();
+
+ AnalyticWindow.Type type = frameUnits == FrameUnitsType.ROWS
+ ? AnalyticWindow.Type.ROWS : AnalyticWindow.Type.RANGE;
+
+ AnalyticWindow.Boundary left = withFrameBoundary(leftBoundary, context);
+ AnalyticWindow.Boundary right = withFrameBoundary(rightBoundary, context);
+
+ return new AnalyticWindow(type, left, right);
+ }
Review Comment:
i think it it better that put translate functions in translator
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] YangShaw commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "YangShaw (via GitHub)" <gi...@apache.org>.
YangShaw commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1092855261
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/NormalizeWindow.java:
##########
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * NormalizeWindow: generate bottomProject for expressions within Window, and topProject for origin output of SQL
+ * e.g. SELECT k1#1, k2#2, SUM(k3#3) OVER (PARTITION BY k4#4 ORDER BY k5#5) FROM t
+ *
+ * Original Plan:
+ * LogicalWindow(
+ * outputs:[k1#1, k2#2, Alias(SUM(k3#3) OVER (PARTITION BY k4#4 ORDER BY k5#5)#6],
+ * windowExpressions:[]
+ * )
+ *
+ * After Normalize:
+ * LogicalProject(k1#1, k2#2, Alias(SlotReference#7)#6)
+ * +-- LogicalWindow(
+ * outputs:[k1#1, k2#2, Alias(SUM(k3#3) OVER (PARTITION BY k4#4 ORDER BY k5#5)#6],
+ * windowExpressions:[Alias(SUM(k3#3) OVER (PARTITION BY k4#4 ORDER BY k5#5)#6]
+ * )
+ * +-- LogicalProject(k1#1, k2#2, k3#3, k4#4, k5#5)
+ *
+ */
+public class NormalizeWindow extends OneRewriteRuleFactory implements NormalizeToSlot {
Review Comment:
found bug, in fixing~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1093164371
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -1730,6 +1853,77 @@ private Optional<DataPartition> toDataPartition(PhysicalDistribute distribute,
}
}
+ private Map<ExprId, SlotRef> getBufferedSlotRefForWindow(WindowFrameGroup windowFrameGroup,
+ PlanTranslatorContext context) {
+ Map<ExprId, SlotRef> bufferedSlotRefForWindow = context.getBufferedSlotRefForWindow();
+ // will be set only once when visiting first PhysicalWindow
Review Comment:
i don't think it is right. for example:
```sql
select
count(1) over (...)
from (
select *
from
a
join (select count(1) over (...) from b) c on a.c1 = c.c1
) d;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] YangShaw commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "YangShaw (via GitHub)" <gi...@apache.org>.
YangShaw commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1095317228
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/algebra/Window.java:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.algebra;
+
+import org.apache.doris.analysis.AnalyticWindow;
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.translator.ExpressionTranslator;
+import org.apache.doris.nereids.glue.translator.PlanTranslatorContext;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.expressions.functions.window.FrameBoundType;
+import org.apache.doris.nereids.trees.expressions.functions.window.FrameBoundary;
+import org.apache.doris.nereids.trees.expressions.functions.window.FrameUnitsType;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+
+import java.math.BigDecimal;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * interface for LogicalWindow and PhysicalWindow
+ */
+public interface Window {
+
+ List<NamedExpression> getOutputExpressions();
+
+ default List<Expression> extractExpressionsFromWindow(List<NamedExpression> windowExpressions) {
+ return windowExpressions.stream().map(expression -> expression.child(0))
+ .map(org.apache.doris.nereids.trees.expressions.Window.class::cast)
+ .flatMap(window -> window.getExpressionsInWindowSpec().stream())
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * translate WindowFrame to AnalyticWindow
+ */
+ default AnalyticWindow translateWindowFrame(WindowFrame windowFrame, PlanTranslatorContext context) {
+ FrameUnitsType frameUnits = windowFrame.getFrameUnits();
+ FrameBoundary leftBoundary = windowFrame.getLeftBoundary();
+ FrameBoundary rightBoundary = windowFrame.getRightBoundary();
+
+ AnalyticWindow.Type type = frameUnits == FrameUnitsType.ROWS
+ ? AnalyticWindow.Type.ROWS : AnalyticWindow.Type.RANGE;
+
+ AnalyticWindow.Boundary left = withFrameBoundary(leftBoundary, context);
+ AnalyticWindow.Boundary right = withFrameBoundary(rightBoundary, context);
+
+ return new AnalyticWindow(type, left, right);
+ }
Review Comment:
due to that AnalyticWindow doesn't extend Expr, it is hard to use visitWindowFrame() in ExpressionTranslator. And in consider of that the code related of translate windowFrame is long, I put these functions in Window interface rather than PhysicalPlanTranslator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1104245323
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/WindowExpression.java:
##########
@@ -0,0 +1,211 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.DataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * represents window function. WindowFunction of this window is saved as Window's child,
+ * which is an UnboundFunction at first and will be analyzed as relevant BoundFunction
+ * (can be a WindowFunction or AggregateFunction) after BindFunction.
+ */
+public class WindowExpression extends Expression implements PropagateNullable {
Review Comment:
```suggestion
public class WindowExpression extends Expression {
@Override
public boolean nullable() {
return function.nullable();
}
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/Ntile.java:
##########
@@ -0,0 +1,82 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.window;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable;
+import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.shape.LeafExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.IntegerType;
+import org.apache.doris.nereids.types.LargeIntType;
+import org.apache.doris.nereids.types.SmallIntType;
+import org.apache.doris.nereids.types.TinyIntType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * window function: Ntile()
+ */
+public class Ntile extends WindowFunction implements LeafExpression, AlwaysNotNullable, ExplicitlyCastableSignature {
+
+ public static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
+ FunctionSignature.ret(BigIntType.INSTANCE).args(TinyIntType.INSTANCE),
+ FunctionSignature.ret(BigIntType.INSTANCE).args(SmallIntType.INSTANCE),
+ FunctionSignature.ret(BigIntType.INSTANCE).args(IntegerType.INSTANCE),
+ FunctionSignature.ret(BigIntType.INSTANCE).args(BigIntType.INSTANCE),
+ FunctionSignature.ret(LargeIntType.INSTANCE).args(LargeIntType.INSTANCE)
+ );
+
+ private Expression buckets;
+
+ public Ntile(Expression buckets) {
+ super("ntile", buckets);
+ this.buckets = buckets;
+ }
+
+ public Expression getBuckets() {
+ return buckets;
+ }
+
+ @Override
+ public Ntile withChildren(List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 1);
+ return new Ntile(children.get(0));
+ }
+
+ @Override
+ public List<FunctionSignature> getSignatures() {
+ return SIGNATURES;
+ }
+
+ @Override
+ public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
+ return visitor.visitNtile(this, context);
+ }
+
+ @Override
+ public DataType getDataType() {
+ return IntegerType.INSTANCE;
+ }
Review Comment:
remove this function
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/DenseRank.java:
##########
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.window;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable;
+import org.apache.doris.nereids.trees.expressions.shape.LeafExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.IntegerType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * Window function: Dense_rank()
+ */
+public class DenseRank extends WindowFunction implements AlwaysNotNullable, LeafExpression {
+
+ public DenseRank() {
+ super("dense_rank");
+ }
+
+ @Override
+ public List<FunctionSignature> getSignatures() {
+ return ImmutableList.of(FunctionSignature.ret(BigIntType.INSTANCE).args());
+ }
+
+ @Override
+ public FunctionSignature searchSignature(List<FunctionSignature> signatures) {
+ return signatures.get(0);
+ }
+
+ @Override
+ public DenseRank withChildren(List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 0);
+ return new DenseRank();
+ }
+
+ @Override
+ public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
+ return visitor.visitDenseRank(this, context);
+ }
+
+ @Override
+ public DataType getDataType() {
+ return IntegerType.INSTANCE;
+ }
Review Comment:
remove this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14397: [feature](nereids)support window function
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #14397:
URL: https://github.com/apache/doris/pull/14397#issuecomment-1366660008
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.69 seconds
load time: 637 seconds
storage size: 17123659552 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221228133821_clickbench_pr_70150.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] YangShaw commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "YangShaw (via GitHub)" <gi...@apache.org>.
YangShaw commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1101077598
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Window.java:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.DataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * represents window function. WindowFunction of this window is saved as Window's child,
+ * which is an UnboundFunction at first and will be analyzed as relevant BoundFunction
+ * (can be a WindowFunction or AggregateFunction) after BindFunction.
+ */
+public class Window extends Expression implements UnaryExpression, PropagateNullable {
+
+ private Optional<List<Expression>> partitionKeyList;
Review Comment:
good idea. I use Set in WindowFrameGroup to save partition keys, which will be used to check compatibility between different windowExpressions and generate requiredProperties. In WindowExpression, I still use List to keep the original keys and order from sql.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] YangShaw commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "YangShaw (via GitHub)" <gi...@apache.org>.
YangShaw commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1094410796
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Window.java:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.DataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * represents window function. WindowFunction of this window is saved as Window's child,
+ * which is an UnboundFunction at first and will be analyzed as relevant BoundFunction
+ * (can be a WindowFunction or AggregateFunction) after BindFunction.
+ */
+public class Window extends Expression implements UnaryExpression, PropagateNullable {
Review Comment:
so stupid what I have ever done..
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/NormalizeWindow.java:
##########
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * NormalizeWindow: generate bottomProject for expressions within Window, and topProject for origin output of SQL
+ * e.g. SELECT k1#1, k2#2, SUM(k3#3) OVER (PARTITION BY k4#4 ORDER BY k5#5) FROM t
+ *
+ * Original Plan:
+ * LogicalWindow(
+ * outputs:[k1#1, k2#2, Alias(SUM(k3#3) OVER (PARTITION BY k4#4 ORDER BY k5#5)#6],
+ * windowExpressions:[]
+ * )
+ *
+ * After Normalize:
+ * LogicalProject(k1#1, k2#2, Alias(SlotReference#7)#6)
+ * +-- LogicalWindow(
+ * outputs:[k1#1, k2#2, Alias(SUM(k3#3) OVER (PARTITION BY k4#4 ORDER BY k5#5)#6],
+ * windowExpressions:[Alias(SUM(k3#3) OVER (PARTITION BY k4#4 ORDER BY k5#5)#6]
+ * )
+ * +-- LogicalProject(k1#1, k2#2, k3#3, k4#4, k5#5)
+ *
+ */
+public class NormalizeWindow extends OneRewriteRuleFactory implements NormalizeToSlot {
Review Comment:
found bug, in fixing~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1092937051
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
+ // todo: partitionByEq and orderByEq?
+ // requiredProperties:
+ // Distribution: partitionKeys
+ // Order: requiredOrderKeys
+ PhysicalWindow physicalWindow = new PhysicalWindow<>(
+ logicalWindow.getOutputExpressions(),
+ logicalWindow.getWindowExpressions(),
+ windowFrameGroup,
+ logicalWindow.getLogicalProperties(),
+ root);
+ // todo: add isAnalyticSort to physicalWindow
+ if (windowFrameGroup.partitionKeyList.isEmpty() && requiredOrderKeys.isEmpty()) {
+ return physicalWindow.withRequirePropertiesAndChild(RequireProperties.followParent(), root);
+ }
+
+ PhysicalProperties properties;
+ if (windowFrameGroup.partitionKeyList.isEmpty()) {
+ properties = new PhysicalProperties(new OrderSpec(requiredOrderKeys));
+ } else {
+ // todo: add new ShuffleType for window, like ShuffleType.WINDOW
+ properties = PhysicalProperties.createHash(
+ windowFrameGroup.partitionKeyList, DistributionSpecHash.ShuffleType.ENFORCED);
+ // requiredOrderKeys contain partitionKeys, so there is no need to check if requiredOrderKeys.isEmpty()
+ properties = properties.withOrderSpec(new OrderSpec(requiredOrderKeys));
+ }
+
+ RequireProperties requireProperties = RequireProperties.of(properties);
+ return physicalWindow.withRequirePropertiesAndChild(requireProperties, root);
+ }
+
+ /* ********************************************************************************************
+ * WindowFunctionRelatedGroups
+ * ******************************************************************************************** */
+
+ // todo: can we simplify the following three algorithms?
+ private List<WindowFrameGroup> createCommonWindowFrameGroups(List<NamedExpression> windowList) {
+ List<WindowFrameGroup> windowFrameGroupList = Lists.newArrayList();
+ for (int i = 0; i < windowList.size(); i++) {
+ NamedExpression windowAlias = windowList.get(i);
+
+ boolean matched = false;
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ if (windowFrameGroup.isCompatible(windowAlias)) {
+ windowFrameGroup.addGroupMember(windowAlias);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ windowFrameGroupList.add(new WindowFrameGroup(windowAlias));
+ }
+ }
+
+ for (WindowFrameGroup wfg : windowFrameGroupList) {
+ wfg.setTupleSize(wfg.groupList.stream().mapToInt(window -> window.child(0).getDataType().width()).sum());
+ }
+
+ return windowFrameGroupList;
+ }
+
+ private List<OrderKeyGroup> createCommonOrderKeyGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ List<OrderKeyGroup> orderKeyGroupList = Lists.newArrayList();
+
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ boolean matched = false;
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ if (orderKeyGroup.isCompatible(windowFrameGroup)) {
+ orderKeyGroup.addGroupMember(windowFrameGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ orderKeyGroupList.add(new OrderKeyGroup(windowFrameGroup));
+ }
+ }
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ okg.setTupleSize(okg.getGroupList().stream().mapToInt(WindowFrameGroup::getTupleSize).sum());
+ }
+
+ return orderKeyGroupList;
+ }
+
+ private List<PartitionKeyGroup> createCommonPartitionKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ List<PartitionKeyGroup> partitionKeyGroupList = Lists.newArrayList();
+
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ boolean matched = false;
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ if (partitionKeyGroup.isCompatible(orderKeyGroup)) {
+ partitionKeyGroup.addGroupMember(orderKeyGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ partitionKeyGroupList.add(new PartitionKeyGroup(orderKeyGroup));
+ }
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ pkg.setTupleSize(pkg.getGroupList().stream().mapToInt(OrderKeyGroup::getTupleSize).sum());
+ }
+
+ return partitionKeyGroupList;
+ }
+
+ private void mergeOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (OrderKeyGroup okg1 : orderKeyGroupList) {
+ for (OrderKeyGroup okg2 : orderKeyGroupList) {
+ if (okg1 != okg2 && okg2.isPrefixOf(okg1)) {
+ // okg2 ∈ okg1
+ okg1.absorb(okg2);
+ orderKeyGroupList.remove(okg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ private void mergePartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (PartitionKeyGroup pkg1 : partitionKeyGroupList) {
+ for (PartitionKeyGroup pkg2 : partitionKeyGroupList) {
+ if (pkg1 != pkg2) {
+ // todo:
+ pkg1.absorb(pkg2);
+ partitionKeyGroupList.remove(pkg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ /**
+ * [This comment references AnalyticPlanner.orderGroups()]
+ *
+ * Order partition groups (and the SortGroups, WindowFrameGroups within them) by increasing
+ * totalOutputTupleSize. This minimizes the total volume of data that needs to be
+ * repartitioned and sorted.
+ *
+ * Always move the non-partitioning partition group to the end, if it exists.
+ */
+ private void sortPartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+
+ PartitionKeyGroup noPartition = null;
+ // after createCommonPartitionKeyGroups(), there will be at most one empty partition.
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ if (pkg.partitionKeyList.isEmpty()) {
+ noPartition = pkg;
+ partitionKeyGroupList.remove(noPartition);
+ break;
+ }
+ }
+
+ if (noPartition != null) {
+ partitionKeyGroupList.remove(noPartition);
+ }
+
+ partitionKeyGroupList.sort((pkg1, pkg2) ->
+ Integer.compare(pkg1.getTupleSize() - pkg2.getTupleSize(), 0)
+ );
+
+ if (noPartition != null) {
+ partitionKeyGroupList.add(noPartition);
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ sortOrderKeyGroups(pkg.getGroupList());
+ }
+
+ }
+
+ private void sortOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ orderKeyGroupList.sort((okg1, okg2) ->
+ Integer.compare(okg1.getTupleSize() - okg2.getTupleSize(), 0)
+ );
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ sortWindowFrameGroups(okg.getGroupList());
+ }
+ }
+
+ private void sortWindowFrameGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ windowFrameGroupList.sort((wfg1, wfg2) ->
+ Integer.compare(wfg1.getTupleSize() - wfg2.getTupleSize(), 0)
+ );
+ }
+
+ /**
+ * Window Functions that have common PartitionKeys, OrderKeys and WindowFrame
+ */
+ public static class WindowFrameGroup extends WindowFunctionRelatedGroup<NamedExpression> {
+
+ private final List<Expression> partitionKeyList;
+ private final List<OrderKey> orderKeyList;
+ private final WindowFrame windowFrame;
+
+ public WindowFrameGroup(NamedExpression windowAlias) {
+ Window window = (Window) (windowAlias.child(0));
+ partitionKeyList = window.getPartitionKeyList().orElse(Lists.newArrayList());
+ orderKeyList = window.getOrderKeyList().orElse(Lists.newArrayList());
Review Comment:
```suggestion
partitionKeyList = window.getPartitionKeyList().orElse(ImmutableList.of());
orderKeyList = window.getOrderKeyList().orElse(ImmutableList.of());
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
+ // todo: partitionByEq and orderByEq?
+ // requiredProperties:
+ // Distribution: partitionKeys
+ // Order: requiredOrderKeys
+ PhysicalWindow physicalWindow = new PhysicalWindow<>(
+ logicalWindow.getOutputExpressions(),
+ logicalWindow.getWindowExpressions(),
+ windowFrameGroup,
+ logicalWindow.getLogicalProperties(),
+ root);
+ // todo: add isAnalyticSort to physicalWindow
+ if (windowFrameGroup.partitionKeyList.isEmpty() && requiredOrderKeys.isEmpty()) {
+ return physicalWindow.withRequirePropertiesAndChild(RequireProperties.followParent(), root);
+ }
+
+ PhysicalProperties properties;
+ if (windowFrameGroup.partitionKeyList.isEmpty()) {
+ properties = new PhysicalProperties(new OrderSpec(requiredOrderKeys));
+ } else {
+ // todo: add new ShuffleType for window, like ShuffleType.WINDOW
+ properties = PhysicalProperties.createHash(
+ windowFrameGroup.partitionKeyList, DistributionSpecHash.ShuffleType.ENFORCED);
+ // requiredOrderKeys contain partitionKeys, so there is no need to check if requiredOrderKeys.isEmpty()
+ properties = properties.withOrderSpec(new OrderSpec(requiredOrderKeys));
+ }
+
+ RequireProperties requireProperties = RequireProperties.of(properties);
+ return physicalWindow.withRequirePropertiesAndChild(requireProperties, root);
+ }
+
+ /* ********************************************************************************************
+ * WindowFunctionRelatedGroups
+ * ******************************************************************************************** */
+
+ // todo: can we simplify the following three algorithms?
+ private List<WindowFrameGroup> createCommonWindowFrameGroups(List<NamedExpression> windowList) {
+ List<WindowFrameGroup> windowFrameGroupList = Lists.newArrayList();
+ for (int i = 0; i < windowList.size(); i++) {
+ NamedExpression windowAlias = windowList.get(i);
+
+ boolean matched = false;
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ if (windowFrameGroup.isCompatible(windowAlias)) {
+ windowFrameGroup.addGroupMember(windowAlias);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ windowFrameGroupList.add(new WindowFrameGroup(windowAlias));
+ }
+ }
+
+ for (WindowFrameGroup wfg : windowFrameGroupList) {
+ wfg.setTupleSize(wfg.groupList.stream().mapToInt(window -> window.child(0).getDataType().width()).sum());
+ }
+
+ return windowFrameGroupList;
+ }
+
+ private List<OrderKeyGroup> createCommonOrderKeyGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ List<OrderKeyGroup> orderKeyGroupList = Lists.newArrayList();
+
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ boolean matched = false;
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ if (orderKeyGroup.isCompatible(windowFrameGroup)) {
+ orderKeyGroup.addGroupMember(windowFrameGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ orderKeyGroupList.add(new OrderKeyGroup(windowFrameGroup));
+ }
+ }
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ okg.setTupleSize(okg.getGroupList().stream().mapToInt(WindowFrameGroup::getTupleSize).sum());
+ }
+
+ return orderKeyGroupList;
+ }
+
+ private List<PartitionKeyGroup> createCommonPartitionKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ List<PartitionKeyGroup> partitionKeyGroupList = Lists.newArrayList();
+
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ boolean matched = false;
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ if (partitionKeyGroup.isCompatible(orderKeyGroup)) {
+ partitionKeyGroup.addGroupMember(orderKeyGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ partitionKeyGroupList.add(new PartitionKeyGroup(orderKeyGroup));
+ }
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ pkg.setTupleSize(pkg.getGroupList().stream().mapToInt(OrderKeyGroup::getTupleSize).sum());
+ }
+
+ return partitionKeyGroupList;
+ }
+
+ private void mergeOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (OrderKeyGroup okg1 : orderKeyGroupList) {
+ for (OrderKeyGroup okg2 : orderKeyGroupList) {
+ if (okg1 != okg2 && okg2.isPrefixOf(okg1)) {
+ // okg2 ∈ okg1
+ okg1.absorb(okg2);
+ orderKeyGroupList.remove(okg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ private void mergePartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (PartitionKeyGroup pkg1 : partitionKeyGroupList) {
+ for (PartitionKeyGroup pkg2 : partitionKeyGroupList) {
+ if (pkg1 != pkg2) {
+ // todo:
+ pkg1.absorb(pkg2);
+ partitionKeyGroupList.remove(pkg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ /**
+ * [This comment references AnalyticPlanner.orderGroups()]
+ *
+ * Order partition groups (and the SortGroups, WindowFrameGroups within them) by increasing
+ * totalOutputTupleSize. This minimizes the total volume of data that needs to be
+ * repartitioned and sorted.
+ *
+ * Always move the non-partitioning partition group to the end, if it exists.
+ */
+ private void sortPartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+
+ PartitionKeyGroup noPartition = null;
+ // after createCommonPartitionKeyGroups(), there will be at most one empty partition.
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ if (pkg.partitionKeyList.isEmpty()) {
+ noPartition = pkg;
+ partitionKeyGroupList.remove(noPartition);
+ break;
+ }
+ }
+
+ if (noPartition != null) {
+ partitionKeyGroupList.remove(noPartition);
+ }
+
+ partitionKeyGroupList.sort((pkg1, pkg2) ->
+ Integer.compare(pkg1.getTupleSize() - pkg2.getTupleSize(), 0)
+ );
+
+ if (noPartition != null) {
+ partitionKeyGroupList.add(noPartition);
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ sortOrderKeyGroups(pkg.getGroupList());
+ }
+
+ }
+
+ private void sortOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ orderKeyGroupList.sort((okg1, okg2) ->
+ Integer.compare(okg1.getTupleSize() - okg2.getTupleSize(), 0)
+ );
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ sortWindowFrameGroups(okg.getGroupList());
+ }
+ }
+
+ private void sortWindowFrameGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ windowFrameGroupList.sort((wfg1, wfg2) ->
+ Integer.compare(wfg1.getTupleSize() - wfg2.getTupleSize(), 0)
+ );
+ }
+
+ /**
+ * Window Functions that have common PartitionKeys, OrderKeys and WindowFrame
+ */
+ public static class WindowFrameGroup extends WindowFunctionRelatedGroup<NamedExpression> {
+
+ private final List<Expression> partitionKeyList;
+ private final List<OrderKey> orderKeyList;
+ private final WindowFrame windowFrame;
+
+ public WindowFrameGroup(NamedExpression windowAlias) {
+ Window window = (Window) (windowAlias.child(0));
+ partitionKeyList = window.getPartitionKeyList().orElse(Lists.newArrayList());
+ orderKeyList = window.getOrderKeyList().orElse(Lists.newArrayList());
+ windowFrame = window.getWindowFrame().orElse(null);
+ groupList.add(windowAlias);
+ }
+
+ @Override
+ public void addGroupMember(NamedExpression windowAlias) {
+ groupList.add(windowAlias);
+ }
+
+ @Override
+ public boolean isCompatible(NamedExpression windowAlias) {
+ // The comparison of PartitionKey is irrelevant to key's order,
+ // but not in OrderKey' comparison.
+ Window window = (Window) (windowAlias.child(0));
+
+ List<Expression> otherPartitionKeyList = window.getPartitionKeyList().orElse(null);
+ List<OrderKey> otherOrderKeyList = window.getOrderKeyList().orElse(null);
+ WindowFrame otherWindowFrame = window.getWindowFrame().orElse(null);
Review Comment:
should not return null in `orElse`
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
Review Comment:
```suggestion
private PhysicalWindow implement(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
+ // todo: partitionByEq and orderByEq?
+ // requiredProperties:
+ // Distribution: partitionKeys
+ // Order: requiredOrderKeys
+ PhysicalWindow physicalWindow = new PhysicalWindow<>(
+ logicalWindow.getOutputExpressions(),
+ logicalWindow.getWindowExpressions(),
+ windowFrameGroup,
+ logicalWindow.getLogicalProperties(),
+ root);
+ // todo: add isAnalyticSort to physicalWindow
+ if (windowFrameGroup.partitionKeyList.isEmpty() && requiredOrderKeys.isEmpty()) {
+ return physicalWindow.withRequirePropertiesAndChild(RequireProperties.followParent(), root);
+ }
+
+ PhysicalProperties properties;
+ if (windowFrameGroup.partitionKeyList.isEmpty()) {
+ properties = new PhysicalProperties(new OrderSpec(requiredOrderKeys));
+ } else {
+ // todo: add new ShuffleType for window, like ShuffleType.WINDOW
+ properties = PhysicalProperties.createHash(
+ windowFrameGroup.partitionKeyList, DistributionSpecHash.ShuffleType.ENFORCED);
+ // requiredOrderKeys contain partitionKeys, so there is no need to check if requiredOrderKeys.isEmpty()
+ properties = properties.withOrderSpec(new OrderSpec(requiredOrderKeys));
+ }
+
+ RequireProperties requireProperties = RequireProperties.of(properties);
+ return physicalWindow.withRequirePropertiesAndChild(requireProperties, root);
+ }
+
+ /* ********************************************************************************************
+ * WindowFunctionRelatedGroups
+ * ******************************************************************************************** */
+
+ // todo: can we simplify the following three algorithms?
+ private List<WindowFrameGroup> createCommonWindowFrameGroups(List<NamedExpression> windowList) {
+ List<WindowFrameGroup> windowFrameGroupList = Lists.newArrayList();
+ for (int i = 0; i < windowList.size(); i++) {
+ NamedExpression windowAlias = windowList.get(i);
+
+ boolean matched = false;
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ if (windowFrameGroup.isCompatible(windowAlias)) {
+ windowFrameGroup.addGroupMember(windowAlias);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ windowFrameGroupList.add(new WindowFrameGroup(windowAlias));
+ }
+ }
+
+ for (WindowFrameGroup wfg : windowFrameGroupList) {
+ wfg.setTupleSize(wfg.groupList.stream().mapToInt(window -> window.child(0).getDataType().width()).sum());
+ }
+
+ return windowFrameGroupList;
+ }
+
+ private List<OrderKeyGroup> createCommonOrderKeyGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ List<OrderKeyGroup> orderKeyGroupList = Lists.newArrayList();
+
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ boolean matched = false;
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ if (orderKeyGroup.isCompatible(windowFrameGroup)) {
+ orderKeyGroup.addGroupMember(windowFrameGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ orderKeyGroupList.add(new OrderKeyGroup(windowFrameGroup));
+ }
+ }
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ okg.setTupleSize(okg.getGroupList().stream().mapToInt(WindowFrameGroup::getTupleSize).sum());
+ }
+
+ return orderKeyGroupList;
+ }
+
+ private List<PartitionKeyGroup> createCommonPartitionKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ List<PartitionKeyGroup> partitionKeyGroupList = Lists.newArrayList();
+
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ boolean matched = false;
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ if (partitionKeyGroup.isCompatible(orderKeyGroup)) {
+ partitionKeyGroup.addGroupMember(orderKeyGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ partitionKeyGroupList.add(new PartitionKeyGroup(orderKeyGroup));
+ }
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ pkg.setTupleSize(pkg.getGroupList().stream().mapToInt(OrderKeyGroup::getTupleSize).sum());
+ }
+
+ return partitionKeyGroupList;
+ }
+
+ private void mergeOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (OrderKeyGroup okg1 : orderKeyGroupList) {
+ for (OrderKeyGroup okg2 : orderKeyGroupList) {
+ if (okg1 != okg2 && okg2.isPrefixOf(okg1)) {
+ // okg2 ∈ okg1
+ okg1.absorb(okg2);
+ orderKeyGroupList.remove(okg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ private void mergePartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (PartitionKeyGroup pkg1 : partitionKeyGroupList) {
+ for (PartitionKeyGroup pkg2 : partitionKeyGroupList) {
+ if (pkg1 != pkg2) {
+ // todo:
+ pkg1.absorb(pkg2);
+ partitionKeyGroupList.remove(pkg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ /**
+ * [This comment references AnalyticPlanner.orderGroups()]
+ *
+ * Order partition groups (and the SortGroups, WindowFrameGroups within them) by increasing
+ * totalOutputTupleSize. This minimizes the total volume of data that needs to be
+ * repartitioned and sorted.
+ *
+ * Always move the non-partitioning partition group to the end, if it exists.
+ */
+ private void sortPartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+
+ PartitionKeyGroup noPartition = null;
+ // after createCommonPartitionKeyGroups(), there will be at most one empty partition.
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ if (pkg.partitionKeyList.isEmpty()) {
+ noPartition = pkg;
+ partitionKeyGroupList.remove(noPartition);
+ break;
+ }
+ }
+
+ if (noPartition != null) {
+ partitionKeyGroupList.remove(noPartition);
+ }
+
+ partitionKeyGroupList.sort((pkg1, pkg2) ->
+ Integer.compare(pkg1.getTupleSize() - pkg2.getTupleSize(), 0)
+ );
+
+ if (noPartition != null) {
+ partitionKeyGroupList.add(noPartition);
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ sortOrderKeyGroups(pkg.getGroupList());
+ }
+
+ }
+
+ private void sortOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ orderKeyGroupList.sort((okg1, okg2) ->
+ Integer.compare(okg1.getTupleSize() - okg2.getTupleSize(), 0)
+ );
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ sortWindowFrameGroups(okg.getGroupList());
+ }
+ }
+
+ private void sortWindowFrameGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ windowFrameGroupList.sort((wfg1, wfg2) ->
+ Integer.compare(wfg1.getTupleSize() - wfg2.getTupleSize(), 0)
+ );
+ }
+
+ /**
+ * Window Functions that have common PartitionKeys, OrderKeys and WindowFrame
+ */
+ public static class WindowFrameGroup extends WindowFunctionRelatedGroup<NamedExpression> {
+
+ private final List<Expression> partitionKeyList;
+ private final List<OrderKey> orderKeyList;
+ private final WindowFrame windowFrame;
+
+ public WindowFrameGroup(NamedExpression windowAlias) {
+ Window window = (Window) (windowAlias.child(0));
+ partitionKeyList = window.getPartitionKeyList().orElse(Lists.newArrayList());
+ orderKeyList = window.getOrderKeyList().orElse(Lists.newArrayList());
+ windowFrame = window.getWindowFrame().orElse(null);
Review Comment:
orElse should return default windowFrame
RANGE BETWEEN UNBOUNDED_PRECEDING AND CURRENT_ROW
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
+ // todo: partitionByEq and orderByEq?
+ // requiredProperties:
+ // Distribution: partitionKeys
+ // Order: requiredOrderKeys
+ PhysicalWindow physicalWindow = new PhysicalWindow<>(
+ logicalWindow.getOutputExpressions(),
+ logicalWindow.getWindowExpressions(),
+ windowFrameGroup,
+ logicalWindow.getLogicalProperties(),
+ root);
+ // todo: add isAnalyticSort to physicalWindow
+ if (windowFrameGroup.partitionKeyList.isEmpty() && requiredOrderKeys.isEmpty()) {
+ return physicalWindow.withRequirePropertiesAndChild(RequireProperties.followParent(), root);
+ }
+
+ PhysicalProperties properties;
+ if (windowFrameGroup.partitionKeyList.isEmpty()) {
+ properties = new PhysicalProperties(new OrderSpec(requiredOrderKeys));
+ } else {
+ // todo: add new ShuffleType for window, like ShuffleType.WINDOW
+ properties = PhysicalProperties.createHash(
+ windowFrameGroup.partitionKeyList, DistributionSpecHash.ShuffleType.ENFORCED);
+ // requiredOrderKeys contain partitionKeys, so there is no need to check if requiredOrderKeys.isEmpty()
+ properties = properties.withOrderSpec(new OrderSpec(requiredOrderKeys));
+ }
+
+ RequireProperties requireProperties = RequireProperties.of(properties);
+ return physicalWindow.withRequirePropertiesAndChild(requireProperties, root);
+ }
+
+ /* ********************************************************************************************
+ * WindowFunctionRelatedGroups
+ * ******************************************************************************************** */
+
+ // todo: can we simplify the following three algorithms?
+ private List<WindowFrameGroup> createCommonWindowFrameGroups(List<NamedExpression> windowList) {
+ List<WindowFrameGroup> windowFrameGroupList = Lists.newArrayList();
+ for (int i = 0; i < windowList.size(); i++) {
+ NamedExpression windowAlias = windowList.get(i);
+
+ boolean matched = false;
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ if (windowFrameGroup.isCompatible(windowAlias)) {
+ windowFrameGroup.addGroupMember(windowAlias);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ windowFrameGroupList.add(new WindowFrameGroup(windowAlias));
+ }
+ }
+
+ for (WindowFrameGroup wfg : windowFrameGroupList) {
+ wfg.setTupleSize(wfg.groupList.stream().mapToInt(window -> window.child(0).getDataType().width()).sum());
+ }
+
+ return windowFrameGroupList;
+ }
+
+ private List<OrderKeyGroup> createCommonOrderKeyGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ List<OrderKeyGroup> orderKeyGroupList = Lists.newArrayList();
+
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ boolean matched = false;
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ if (orderKeyGroup.isCompatible(windowFrameGroup)) {
+ orderKeyGroup.addGroupMember(windowFrameGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ orderKeyGroupList.add(new OrderKeyGroup(windowFrameGroup));
+ }
+ }
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ okg.setTupleSize(okg.getGroupList().stream().mapToInt(WindowFrameGroup::getTupleSize).sum());
+ }
+
+ return orderKeyGroupList;
+ }
+
+ private List<PartitionKeyGroup> createCommonPartitionKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ List<PartitionKeyGroup> partitionKeyGroupList = Lists.newArrayList();
+
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ boolean matched = false;
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ if (partitionKeyGroup.isCompatible(orderKeyGroup)) {
+ partitionKeyGroup.addGroupMember(orderKeyGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ partitionKeyGroupList.add(new PartitionKeyGroup(orderKeyGroup));
+ }
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ pkg.setTupleSize(pkg.getGroupList().stream().mapToInt(OrderKeyGroup::getTupleSize).sum());
+ }
+
+ return partitionKeyGroupList;
+ }
+
+ private void mergeOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (OrderKeyGroup okg1 : orderKeyGroupList) {
+ for (OrderKeyGroup okg2 : orderKeyGroupList) {
+ if (okg1 != okg2 && okg2.isPrefixOf(okg1)) {
+ // okg2 ∈ okg1
+ okg1.absorb(okg2);
+ orderKeyGroupList.remove(okg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ private void mergePartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (PartitionKeyGroup pkg1 : partitionKeyGroupList) {
+ for (PartitionKeyGroup pkg2 : partitionKeyGroupList) {
+ if (pkg1 != pkg2) {
+ // todo:
+ pkg1.absorb(pkg2);
+ partitionKeyGroupList.remove(pkg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ /**
+ * [This comment references AnalyticPlanner.orderGroups()]
+ *
+ * Order partition groups (and the SortGroups, WindowFrameGroups within them) by increasing
+ * totalOutputTupleSize. This minimizes the total volume of data that needs to be
+ * repartitioned and sorted.
+ *
+ * Always move the non-partitioning partition group to the end, if it exists.
+ */
+ private void sortPartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+
+ PartitionKeyGroup noPartition = null;
+ // after createCommonPartitionKeyGroups(), there will be at most one empty partition.
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ if (pkg.partitionKeyList.isEmpty()) {
+ noPartition = pkg;
+ partitionKeyGroupList.remove(noPartition);
+ break;
+ }
+ }
+
+ if (noPartition != null) {
+ partitionKeyGroupList.remove(noPartition);
+ }
+
+ partitionKeyGroupList.sort((pkg1, pkg2) ->
+ Integer.compare(pkg1.getTupleSize() - pkg2.getTupleSize(), 0)
+ );
+
+ if (noPartition != null) {
+ partitionKeyGroupList.add(noPartition);
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ sortOrderKeyGroups(pkg.getGroupList());
+ }
+
+ }
+
+ private void sortOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ orderKeyGroupList.sort((okg1, okg2) ->
+ Integer.compare(okg1.getTupleSize() - okg2.getTupleSize(), 0)
+ );
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ sortWindowFrameGroups(okg.getGroupList());
+ }
+ }
+
+ private void sortWindowFrameGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ windowFrameGroupList.sort((wfg1, wfg2) ->
+ Integer.compare(wfg1.getTupleSize() - wfg2.getTupleSize(), 0)
+ );
+ }
+
+ /**
+ * Window Functions that have common PartitionKeys, OrderKeys and WindowFrame
+ */
+ public static class WindowFrameGroup extends WindowFunctionRelatedGroup<NamedExpression> {
+
+ private final List<Expression> partitionKeyList;
+ private final List<OrderKey> orderKeyList;
+ private final WindowFrame windowFrame;
+
+ public WindowFrameGroup(NamedExpression windowAlias) {
+ Window window = (Window) (windowAlias.child(0));
+ partitionKeyList = window.getPartitionKeyList().orElse(Lists.newArrayList());
+ orderKeyList = window.getOrderKeyList().orElse(Lists.newArrayList());
+ windowFrame = window.getWindowFrame().orElse(null);
+ groupList.add(windowAlias);
+ }
+
+ @Override
+ public void addGroupMember(NamedExpression windowAlias) {
+ groupList.add(windowAlias);
+ }
+
+ @Override
+ public boolean isCompatible(NamedExpression windowAlias) {
+ // The comparison of PartitionKey is irrelevant to key's order,
+ // but not in OrderKey' comparison.
+ Window window = (Window) (windowAlias.child(0));
+
+ List<Expression> otherPartitionKeyList = window.getPartitionKeyList().orElse(null);
+ List<OrderKey> otherOrderKeyList = window.getOrderKeyList().orElse(null);
+ WindowFrame otherWindowFrame = window.getWindowFrame().orElse(null);
+
+ // for PartitionKeys, we don't care about the order of each key, so we use isEqualCollection() to compare
+ // whether these two lists have same keys; but for OrderKeys, the order of each key also make sense, so
+ // we use equals() to compare both the elements and their order in these two lists.
+ if (CollectionUtils.isEqualCollection(partitionKeyList, otherPartitionKeyList)
+ && orderKeyList.equals(otherOrderKeyList)) {
+ // CollectionUtils.isEqualCollection() is absolutely equals to Expr.equalSets()
+ if ((windowFrame == null && otherWindowFrame == null) || windowFrame.equals(otherWindowFrame)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ sb.append("PartitionKeys: ").append(partitionKeyList.stream()
+ .map(Expression::toString)
+ .collect(Collectors.joining(", ", "", ", ")));
+
+ sb.append("OrderKeys: ").append(orderKeyList.stream()
+ .map(OrderKey::toString)
+ .collect(Collectors.joining(", ", "", ", ")));
+
+ sb.append("WindowFrame: ").append(windowFrame);
+ return sb.toString();
+ }
+
+ public List<Expression> getPartitionKeyList() {
+ return partitionKeyList;
+ }
+
+ public List<OrderKey> getOrderKeyList() {
+ return orderKeyList;
+ }
+
+ public WindowFrame getWindowFrame() {
+ return windowFrame;
+ }
+
+ }
+
+ /**
+ * Window Functions that have common PartitionKeys and OrderKeys.
+ */
+ private static class OrderKeyGroup extends WindowFunctionRelatedGroup<WindowFrameGroup> {
+
+ private final List<Expression> partitionKeyList;
+ private final List<OrderKey> orderKeyList;
+
+ public OrderKeyGroup(WindowFrameGroup windowFrameGroup) {
+ partitionKeyList = windowFrameGroup.partitionKeyList;
+ orderKeyList = windowFrameGroup.orderKeyList;
+ groupList.add(windowFrameGroup);
+ }
+
+ @Override
+ public boolean isCompatible(WindowFrameGroup windowFrameGroup) {
+ return CollectionUtils.isEqualCollection(partitionKeyList, windowFrameGroup.partitionKeyList)
+ && orderKeyList.equals(windowFrameGroup.orderKeyList);
+ }
+
+ /**
+ * check if this okg isPrefixOf other, which means that other okg can cover this
+ * e.g. (a, b) is a prefix of (a, b, c), but is NOT a prefix of (b, a, c) or (b, c)
+ */
+ public boolean isPrefixOf(OrderKeyGroup otherOkg) {
+ if (orderKeyList.size() > otherOkg.orderKeyList.size()) {
+ return false;
+ }
+ if (!CollectionUtils.isEqualCollection(partitionKeyList, otherOkg.partitionKeyList)) {
+ return false;
+ }
+
+ for (int i = 0; i < orderKeyList.size(); i++) {
+ if (!orderKeyList.get(i).equals(otherOkg.orderKeyList.get(i))) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * add all of otherOkg's WindowFrameGroups to this groupList
+ */
+ public void absorb(OrderKeyGroup otherOkg) {
+ groupList.addAll(otherOkg.groupList);
+ }
+
+ }
+
+ /**
+ * Window Functions that have common PartitionKeys.
+ */
+ private static class PartitionKeyGroup extends WindowFunctionRelatedGroup<OrderKeyGroup> {
+ public List<Expression> partitionKeyList;
+
+ public PartitionKeyGroup(OrderKeyGroup orderKeyGroup) {
+ partitionKeyList = orderKeyGroup.partitionKeyList;
+ groupList.add(orderKeyGroup);
+ }
+
+ @Override
+ public boolean isCompatible(OrderKeyGroup orderKeyGroup) {
+ return CollectionUtils.isEqualCollection(partitionKeyList, orderKeyGroup.partitionKeyList);
+ }
+
+ /**
+ * absorb other into this:
+ * - partitionKeyList will be the intersection of this two group
+ * - groupList(orderKeyGroup) will be the union of this two group
+ */
+ public void absorb(PartitionKeyGroup otherPkg) {
+ List<Expression> intersectPartitionKeys = partitionKeyList.stream()
+ .filter(expression -> otherPkg.partitionKeyList.contains(expression))
+ .collect(Collectors.toList());
+ partitionKeyList = intersectPartitionKeys;
+ groupList.addAll(otherPkg.groupList);
+ }
+ }
+
+ private abstract static class WindowFunctionRelatedGroup<G> {
+
+ protected List<G> groupList = Lists.newArrayList();
Review Comment:
```suggestion
protected List<G> groups = Lists.newArrayList();
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Window.java:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.DataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * represents window function. WindowFunction of this window is saved as Window's child,
+ * which is an UnboundFunction at first and will be analyzed as relevant BoundFunction
+ * (can be a WindowFunction or AggregateFunction) after BindFunction.
+ */
+public class Window extends Expression implements UnaryExpression, PropagateNullable {
+
+ private Optional<List<Expression>> partitionKeyList;
+
+ private Optional<List<OrderKey>> orderKeyList;
Review Comment:
since order by `a, b, a, c` is same as order by `a, b, c`, so maybe we could add a rule optimize it with LogicalSort togather later.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Window.java:
##########
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.exceptions.UnboundException;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.DataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/**
+ * represents window function. WindowFunction of this window is saved as Window's child,
+ * which is an UnboundFunction at first and will be analyzed as relevant BoundFunction
+ * (can be a WindowFunction or AggregateFunction) after BindFunction.
+ */
+public class Window extends Expression implements UnaryExpression, PropagateNullable {
+
+ private Optional<List<Expression>> partitionKeyList;
Review Comment:
since partition by `a, b, a` equals partition by `b, a`. so could we use a set to save it?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
+ // todo: partitionByEq and orderByEq?
+ // requiredProperties:
+ // Distribution: partitionKeys
+ // Order: requiredOrderKeys
+ PhysicalWindow physicalWindow = new PhysicalWindow<>(
+ logicalWindow.getOutputExpressions(),
+ logicalWindow.getWindowExpressions(),
+ windowFrameGroup,
+ logicalWindow.getLogicalProperties(),
+ root);
+ // todo: add isAnalyticSort to physicalWindow
+ if (windowFrameGroup.partitionKeyList.isEmpty() && requiredOrderKeys.isEmpty()) {
+ return physicalWindow.withRequirePropertiesAndChild(RequireProperties.followParent(), root);
+ }
+
+ PhysicalProperties properties;
+ if (windowFrameGroup.partitionKeyList.isEmpty()) {
+ properties = new PhysicalProperties(new OrderSpec(requiredOrderKeys));
+ } else {
+ // todo: add new ShuffleType for window, like ShuffleType.WINDOW
+ properties = PhysicalProperties.createHash(
+ windowFrameGroup.partitionKeyList, DistributionSpecHash.ShuffleType.ENFORCED);
+ // requiredOrderKeys contain partitionKeys, so there is no need to check if requiredOrderKeys.isEmpty()
+ properties = properties.withOrderSpec(new OrderSpec(requiredOrderKeys));
+ }
+
+ RequireProperties requireProperties = RequireProperties.of(properties);
+ return physicalWindow.withRequirePropertiesAndChild(requireProperties, root);
+ }
+
+ /* ********************************************************************************************
+ * WindowFunctionRelatedGroups
+ * ******************************************************************************************** */
+
+ // todo: can we simplify the following three algorithms?
+ private List<WindowFrameGroup> createCommonWindowFrameGroups(List<NamedExpression> windowList) {
+ List<WindowFrameGroup> windowFrameGroupList = Lists.newArrayList();
+ for (int i = 0; i < windowList.size(); i++) {
+ NamedExpression windowAlias = windowList.get(i);
+
+ boolean matched = false;
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ if (windowFrameGroup.isCompatible(windowAlias)) {
+ windowFrameGroup.addGroupMember(windowAlias);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ windowFrameGroupList.add(new WindowFrameGroup(windowAlias));
+ }
+ }
+
+ for (WindowFrameGroup wfg : windowFrameGroupList) {
+ wfg.setTupleSize(wfg.groupList.stream().mapToInt(window -> window.child(0).getDataType().width()).sum());
+ }
+
+ return windowFrameGroupList;
+ }
+
+ private List<OrderKeyGroup> createCommonOrderKeyGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ List<OrderKeyGroup> orderKeyGroupList = Lists.newArrayList();
+
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ boolean matched = false;
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ if (orderKeyGroup.isCompatible(windowFrameGroup)) {
+ orderKeyGroup.addGroupMember(windowFrameGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ orderKeyGroupList.add(new OrderKeyGroup(windowFrameGroup));
+ }
+ }
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ okg.setTupleSize(okg.getGroupList().stream().mapToInt(WindowFrameGroup::getTupleSize).sum());
+ }
+
+ return orderKeyGroupList;
+ }
+
+ private List<PartitionKeyGroup> createCommonPartitionKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ List<PartitionKeyGroup> partitionKeyGroupList = Lists.newArrayList();
+
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ boolean matched = false;
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ if (partitionKeyGroup.isCompatible(orderKeyGroup)) {
+ partitionKeyGroup.addGroupMember(orderKeyGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ partitionKeyGroupList.add(new PartitionKeyGroup(orderKeyGroup));
+ }
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ pkg.setTupleSize(pkg.getGroupList().stream().mapToInt(OrderKeyGroup::getTupleSize).sum());
+ }
+
+ return partitionKeyGroupList;
+ }
+
+ private void mergeOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (OrderKeyGroup okg1 : orderKeyGroupList) {
+ for (OrderKeyGroup okg2 : orderKeyGroupList) {
+ if (okg1 != okg2 && okg2.isPrefixOf(okg1)) {
+ // okg2 ∈ okg1
+ okg1.absorb(okg2);
+ orderKeyGroupList.remove(okg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ private void mergePartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (PartitionKeyGroup pkg1 : partitionKeyGroupList) {
+ for (PartitionKeyGroup pkg2 : partitionKeyGroupList) {
+ if (pkg1 != pkg2) {
+ // todo:
+ pkg1.absorb(pkg2);
+ partitionKeyGroupList.remove(pkg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ /**
+ * [This comment references AnalyticPlanner.orderGroups()]
+ *
+ * Order partition groups (and the SortGroups, WindowFrameGroups within them) by increasing
+ * totalOutputTupleSize. This minimizes the total volume of data that needs to be
+ * repartitioned and sorted.
+ *
+ * Always move the non-partitioning partition group to the end, if it exists.
+ */
+ private void sortPartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+
+ PartitionKeyGroup noPartition = null;
+ // after createCommonPartitionKeyGroups(), there will be at most one empty partition.
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ if (pkg.partitionKeyList.isEmpty()) {
+ noPartition = pkg;
+ partitionKeyGroupList.remove(noPartition);
+ break;
+ }
+ }
+
+ if (noPartition != null) {
+ partitionKeyGroupList.remove(noPartition);
+ }
+
+ partitionKeyGroupList.sort((pkg1, pkg2) ->
+ Integer.compare(pkg1.getTupleSize() - pkg2.getTupleSize(), 0)
+ );
+
+ if (noPartition != null) {
+ partitionKeyGroupList.add(noPartition);
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ sortOrderKeyGroups(pkg.getGroupList());
+ }
+
+ }
+
+ private void sortOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ orderKeyGroupList.sort((okg1, okg2) ->
+ Integer.compare(okg1.getTupleSize() - okg2.getTupleSize(), 0)
+ );
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ sortWindowFrameGroups(okg.getGroupList());
+ }
+ }
+
+ private void sortWindowFrameGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ windowFrameGroupList.sort((wfg1, wfg2) ->
+ Integer.compare(wfg1.getTupleSize() - wfg2.getTupleSize(), 0)
+ );
+ }
+
+ /**
+ * Window Functions that have common PartitionKeys, OrderKeys and WindowFrame
+ */
+ public static class WindowFrameGroup extends WindowFunctionRelatedGroup<NamedExpression> {
+
+ private final List<Expression> partitionKeyList;
+ private final List<OrderKey> orderKeyList;
Review Comment:
```suggestion
private final List<Expression> partitionKeys;
private final List<OrderKey> orderKeys;
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
+ // todo: partitionByEq and orderByEq?
+ // requiredProperties:
+ // Distribution: partitionKeys
+ // Order: requiredOrderKeys
+ PhysicalWindow physicalWindow = new PhysicalWindow<>(
+ logicalWindow.getOutputExpressions(),
+ logicalWindow.getWindowExpressions(),
+ windowFrameGroup,
+ logicalWindow.getLogicalProperties(),
+ root);
+ // todo: add isAnalyticSort to physicalWindow
+ if (windowFrameGroup.partitionKeyList.isEmpty() && requiredOrderKeys.isEmpty()) {
+ return physicalWindow.withRequirePropertiesAndChild(RequireProperties.followParent(), root);
+ }
+
+ PhysicalProperties properties;
+ if (windowFrameGroup.partitionKeyList.isEmpty()) {
+ properties = new PhysicalProperties(new OrderSpec(requiredOrderKeys));
+ } else {
+ // todo: add new ShuffleType for window, like ShuffleType.WINDOW
+ properties = PhysicalProperties.createHash(
+ windowFrameGroup.partitionKeyList, DistributionSpecHash.ShuffleType.ENFORCED);
+ // requiredOrderKeys contain partitionKeys, so there is no need to check if requiredOrderKeys.isEmpty()
+ properties = properties.withOrderSpec(new OrderSpec(requiredOrderKeys));
+ }
+
+ RequireProperties requireProperties = RequireProperties.of(properties);
+ return physicalWindow.withRequirePropertiesAndChild(requireProperties, root);
+ }
+
+ /* ********************************************************************************************
+ * WindowFunctionRelatedGroups
+ * ******************************************************************************************** */
+
+ // todo: can we simplify the following three algorithms?
+ private List<WindowFrameGroup> createCommonWindowFrameGroups(List<NamedExpression> windowList) {
+ List<WindowFrameGroup> windowFrameGroupList = Lists.newArrayList();
+ for (int i = 0; i < windowList.size(); i++) {
+ NamedExpression windowAlias = windowList.get(i);
+
+ boolean matched = false;
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ if (windowFrameGroup.isCompatible(windowAlias)) {
+ windowFrameGroup.addGroupMember(windowAlias);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ windowFrameGroupList.add(new WindowFrameGroup(windowAlias));
+ }
+ }
+
+ for (WindowFrameGroup wfg : windowFrameGroupList) {
+ wfg.setTupleSize(wfg.groupList.stream().mapToInt(window -> window.child(0).getDataType().width()).sum());
+ }
+
+ return windowFrameGroupList;
+ }
+
+ private List<OrderKeyGroup> createCommonOrderKeyGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ List<OrderKeyGroup> orderKeyGroupList = Lists.newArrayList();
+
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ boolean matched = false;
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ if (orderKeyGroup.isCompatible(windowFrameGroup)) {
+ orderKeyGroup.addGroupMember(windowFrameGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ orderKeyGroupList.add(new OrderKeyGroup(windowFrameGroup));
+ }
+ }
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ okg.setTupleSize(okg.getGroupList().stream().mapToInt(WindowFrameGroup::getTupleSize).sum());
+ }
+
+ return orderKeyGroupList;
+ }
+
+ private List<PartitionKeyGroup> createCommonPartitionKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ List<PartitionKeyGroup> partitionKeyGroupList = Lists.newArrayList();
+
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ boolean matched = false;
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ if (partitionKeyGroup.isCompatible(orderKeyGroup)) {
+ partitionKeyGroup.addGroupMember(orderKeyGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ partitionKeyGroupList.add(new PartitionKeyGroup(orderKeyGroup));
+ }
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ pkg.setTupleSize(pkg.getGroupList().stream().mapToInt(OrderKeyGroup::getTupleSize).sum());
+ }
+
+ return partitionKeyGroupList;
+ }
+
+ private void mergeOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (OrderKeyGroup okg1 : orderKeyGroupList) {
+ for (OrderKeyGroup okg2 : orderKeyGroupList) {
+ if (okg1 != okg2 && okg2.isPrefixOf(okg1)) {
+ // okg2 ∈ okg1
+ okg1.absorb(okg2);
+ orderKeyGroupList.remove(okg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ private void mergePartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (PartitionKeyGroup pkg1 : partitionKeyGroupList) {
+ for (PartitionKeyGroup pkg2 : partitionKeyGroupList) {
+ if (pkg1 != pkg2) {
+ // todo:
+ pkg1.absorb(pkg2);
+ partitionKeyGroupList.remove(pkg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ /**
+ * [This comment references AnalyticPlanner.orderGroups()]
+ *
+ * Order partition groups (and the SortGroups, WindowFrameGroups within them) by increasing
+ * totalOutputTupleSize. This minimizes the total volume of data that needs to be
+ * repartitioned and sorted.
+ *
+ * Always move the non-partitioning partition group to the end, if it exists.
+ */
+ private void sortPartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+
+ PartitionKeyGroup noPartition = null;
+ // after createCommonPartitionKeyGroups(), there will be at most one empty partition.
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ if (pkg.partitionKeyList.isEmpty()) {
+ noPartition = pkg;
+ partitionKeyGroupList.remove(noPartition);
+ break;
+ }
+ }
+
+ if (noPartition != null) {
+ partitionKeyGroupList.remove(noPartition);
+ }
+
+ partitionKeyGroupList.sort((pkg1, pkg2) ->
+ Integer.compare(pkg1.getTupleSize() - pkg2.getTupleSize(), 0)
+ );
+
+ if (noPartition != null) {
+ partitionKeyGroupList.add(noPartition);
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ sortOrderKeyGroups(pkg.getGroupList());
+ }
+
+ }
+
+ private void sortOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ orderKeyGroupList.sort((okg1, okg2) ->
+ Integer.compare(okg1.getTupleSize() - okg2.getTupleSize(), 0)
+ );
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ sortWindowFrameGroups(okg.getGroupList());
+ }
+ }
+
+ private void sortWindowFrameGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ windowFrameGroupList.sort((wfg1, wfg2) ->
+ Integer.compare(wfg1.getTupleSize() - wfg2.getTupleSize(), 0)
+ );
+ }
+
+ /**
+ * Window Functions that have common PartitionKeys, OrderKeys and WindowFrame
+ */
+ public static class WindowFrameGroup extends WindowFunctionRelatedGroup<NamedExpression> {
+
+ private final List<Expression> partitionKeyList;
+ private final List<OrderKey> orderKeyList;
+ private final WindowFrame windowFrame;
+
+ public WindowFrameGroup(NamedExpression windowAlias) {
+ Window window = (Window) (windowAlias.child(0));
+ partitionKeyList = window.getPartitionKeyList().orElse(Lists.newArrayList());
+ orderKeyList = window.getOrderKeyList().orElse(Lists.newArrayList());
+ windowFrame = window.getWindowFrame().orElse(null);
+ groupList.add(windowAlias);
+ }
+
+ @Override
+ public void addGroupMember(NamedExpression windowAlias) {
+ groupList.add(windowAlias);
+ }
+
+ @Override
+ public boolean isCompatible(NamedExpression windowAlias) {
+ // The comparison of PartitionKey is irrelevant to key's order,
+ // but not in OrderKey' comparison.
+ Window window = (Window) (windowAlias.child(0));
+
+ List<Expression> otherPartitionKeyList = window.getPartitionKeyList().orElse(null);
+ List<OrderKey> otherOrderKeyList = window.getOrderKeyList().orElse(null);
+ WindowFrame otherWindowFrame = window.getWindowFrame().orElse(null);
+
+ // for PartitionKeys, we don't care about the order of each key, so we use isEqualCollection() to compare
+ // whether these two lists have same keys; but for OrderKeys, the order of each key also make sense, so
+ // we use equals() to compare both the elements and their order in these two lists.
+ if (CollectionUtils.isEqualCollection(partitionKeyList, otherPartitionKeyList)
+ && orderKeyList.equals(otherOrderKeyList)) {
+ // CollectionUtils.isEqualCollection() is absolutely equals to Expr.equalSets()
+ if ((windowFrame == null && otherWindowFrame == null) || windowFrame.equals(otherWindowFrame)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ sb.append("PartitionKeys: ").append(partitionKeyList.stream()
+ .map(Expression::toString)
+ .collect(Collectors.joining(", ", "", ", ")));
+
+ sb.append("OrderKeys: ").append(orderKeyList.stream()
+ .map(OrderKey::toString)
+ .collect(Collectors.joining(", ", "", ", ")));
+
+ sb.append("WindowFrame: ").append(windowFrame);
+ return sb.toString();
+ }
+
+ public List<Expression> getPartitionKeyList() {
+ return partitionKeyList;
+ }
+
+ public List<OrderKey> getOrderKeyList() {
+ return orderKeyList;
+ }
+
+ public WindowFrame getWindowFrame() {
+ return windowFrame;
+ }
+
+ }
+
+ /**
+ * Window Functions that have common PartitionKeys and OrderKeys.
+ */
+ private static class OrderKeyGroup extends WindowFunctionRelatedGroup<WindowFrameGroup> {
+
+ private final List<Expression> partitionKeyList;
+ private final List<OrderKey> orderKeyList;
+
+ public OrderKeyGroup(WindowFrameGroup windowFrameGroup) {
+ partitionKeyList = windowFrameGroup.partitionKeyList;
+ orderKeyList = windowFrameGroup.orderKeyList;
+ groupList.add(windowFrameGroup);
+ }
+
+ @Override
+ public boolean isCompatible(WindowFrameGroup windowFrameGroup) {
+ return CollectionUtils.isEqualCollection(partitionKeyList, windowFrameGroup.partitionKeyList)
+ && orderKeyList.equals(windowFrameGroup.orderKeyList);
+ }
+
+ /**
+ * check if this okg isPrefixOf other, which means that other okg can cover this
+ * e.g. (a, b) is a prefix of (a, b, c), but is NOT a prefix of (b, a, c) or (b, c)
+ */
+ public boolean isPrefixOf(OrderKeyGroup otherOkg) {
+ if (orderKeyList.size() > otherOkg.orderKeyList.size()) {
+ return false;
+ }
+ if (!CollectionUtils.isEqualCollection(partitionKeyList, otherOkg.partitionKeyList)) {
+ return false;
+ }
+
+ for (int i = 0; i < orderKeyList.size(); i++) {
+ if (!orderKeyList.get(i).equals(otherOkg.orderKeyList.get(i))) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * add all of otherOkg's WindowFrameGroups to this groupList
+ */
+ public void absorb(OrderKeyGroup otherOkg) {
+ groupList.addAll(otherOkg.groupList);
+ }
+
+ }
+
+ /**
+ * Window Functions that have common PartitionKeys.
+ */
+ private static class PartitionKeyGroup extends WindowFunctionRelatedGroup<OrderKeyGroup> {
+ public List<Expression> partitionKeyList;
+
+ public PartitionKeyGroup(OrderKeyGroup orderKeyGroup) {
+ partitionKeyList = orderKeyGroup.partitionKeyList;
+ groupList.add(orderKeyGroup);
+ }
+
+ @Override
+ public boolean isCompatible(OrderKeyGroup orderKeyGroup) {
+ return CollectionUtils.isEqualCollection(partitionKeyList, orderKeyGroup.partitionKeyList);
+ }
+
+ /**
+ * absorb other into this:
+ * - partitionKeyList will be the intersection of this two group
+ * - groupList(orderKeyGroup) will be the union of this two group
+ */
+ public void absorb(PartitionKeyGroup otherPkg) {
+ List<Expression> intersectPartitionKeys = partitionKeyList.stream()
+ .filter(expression -> otherPkg.partitionKeyList.contains(expression))
+ .collect(Collectors.toList());
+ partitionKeyList = intersectPartitionKeys;
+ groupList.addAll(otherPkg.groupList);
+ }
+ }
+
+ private abstract static class WindowFunctionRelatedGroup<G> {
+
+ protected List<G> groupList = Lists.newArrayList();
+
+ protected int tupleSize;
+
+ public abstract boolean isCompatible(G group);
+
+ public void addGroupMember(G group) {
Review Comment:
```suggestion
public void addGroup(G group) {
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
+ // todo: partitionByEq and orderByEq?
+ // requiredProperties:
+ // Distribution: partitionKeys
+ // Order: requiredOrderKeys
+ PhysicalWindow physicalWindow = new PhysicalWindow<>(
+ logicalWindow.getOutputExpressions(),
+ logicalWindow.getWindowExpressions(),
+ windowFrameGroup,
+ logicalWindow.getLogicalProperties(),
+ root);
+ // todo: add isAnalyticSort to physicalWindow
+ if (windowFrameGroup.partitionKeyList.isEmpty() && requiredOrderKeys.isEmpty()) {
+ return physicalWindow.withRequirePropertiesAndChild(RequireProperties.followParent(), root);
+ }
+
+ PhysicalProperties properties;
+ if (windowFrameGroup.partitionKeyList.isEmpty()) {
+ properties = new PhysicalProperties(new OrderSpec(requiredOrderKeys));
+ } else {
+ // todo: add new ShuffleType for window, like ShuffleType.WINDOW
+ properties = PhysicalProperties.createHash(
+ windowFrameGroup.partitionKeyList, DistributionSpecHash.ShuffleType.ENFORCED);
+ // requiredOrderKeys contain partitionKeys, so there is no need to check if requiredOrderKeys.isEmpty()
+ properties = properties.withOrderSpec(new OrderSpec(requiredOrderKeys));
+ }
+
+ RequireProperties requireProperties = RequireProperties.of(properties);
+ return physicalWindow.withRequirePropertiesAndChild(requireProperties, root);
+ }
+
+ /* ********************************************************************************************
+ * WindowFunctionRelatedGroups
+ * ******************************************************************************************** */
+
+ // todo: can we simplify the following three algorithms?
+ private List<WindowFrameGroup> createCommonWindowFrameGroups(List<NamedExpression> windowList) {
+ List<WindowFrameGroup> windowFrameGroupList = Lists.newArrayList();
+ for (int i = 0; i < windowList.size(); i++) {
+ NamedExpression windowAlias = windowList.get(i);
+
+ boolean matched = false;
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ if (windowFrameGroup.isCompatible(windowAlias)) {
+ windowFrameGroup.addGroupMember(windowAlias);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ windowFrameGroupList.add(new WindowFrameGroup(windowAlias));
+ }
+ }
+
+ for (WindowFrameGroup wfg : windowFrameGroupList) {
+ wfg.setTupleSize(wfg.groupList.stream().mapToInt(window -> window.child(0).getDataType().width()).sum());
+ }
+
+ return windowFrameGroupList;
+ }
+
+ private List<OrderKeyGroup> createCommonOrderKeyGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ List<OrderKeyGroup> orderKeyGroupList = Lists.newArrayList();
+
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ boolean matched = false;
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ if (orderKeyGroup.isCompatible(windowFrameGroup)) {
+ orderKeyGroup.addGroupMember(windowFrameGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ orderKeyGroupList.add(new OrderKeyGroup(windowFrameGroup));
+ }
+ }
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ okg.setTupleSize(okg.getGroupList().stream().mapToInt(WindowFrameGroup::getTupleSize).sum());
+ }
+
+ return orderKeyGroupList;
+ }
+
+ private List<PartitionKeyGroup> createCommonPartitionKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ List<PartitionKeyGroup> partitionKeyGroupList = Lists.newArrayList();
+
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ boolean matched = false;
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ if (partitionKeyGroup.isCompatible(orderKeyGroup)) {
+ partitionKeyGroup.addGroupMember(orderKeyGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ partitionKeyGroupList.add(new PartitionKeyGroup(orderKeyGroup));
+ }
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ pkg.setTupleSize(pkg.getGroupList().stream().mapToInt(OrderKeyGroup::getTupleSize).sum());
+ }
+
+ return partitionKeyGroupList;
+ }
+
+ private void mergeOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (OrderKeyGroup okg1 : orderKeyGroupList) {
+ for (OrderKeyGroup okg2 : orderKeyGroupList) {
+ if (okg1 != okg2 && okg2.isPrefixOf(okg1)) {
+ // okg2 ∈ okg1
+ okg1.absorb(okg2);
+ orderKeyGroupList.remove(okg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
Review Comment:
should we need to update tuple size after merge?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
+ // todo: partitionByEq and orderByEq?
+ // requiredProperties:
+ // Distribution: partitionKeys
+ // Order: requiredOrderKeys
+ PhysicalWindow physicalWindow = new PhysicalWindow<>(
+ logicalWindow.getOutputExpressions(),
+ logicalWindow.getWindowExpressions(),
+ windowFrameGroup,
+ logicalWindow.getLogicalProperties(),
+ root);
+ // todo: add isAnalyticSort to physicalWindow
+ if (windowFrameGroup.partitionKeyList.isEmpty() && requiredOrderKeys.isEmpty()) {
+ return physicalWindow.withRequirePropertiesAndChild(RequireProperties.followParent(), root);
+ }
+
+ PhysicalProperties properties;
+ if (windowFrameGroup.partitionKeyList.isEmpty()) {
+ properties = new PhysicalProperties(new OrderSpec(requiredOrderKeys));
+ } else {
+ // todo: add new ShuffleType for window, like ShuffleType.WINDOW
+ properties = PhysicalProperties.createHash(
+ windowFrameGroup.partitionKeyList, DistributionSpecHash.ShuffleType.ENFORCED);
+ // requiredOrderKeys contain partitionKeys, so there is no need to check if requiredOrderKeys.isEmpty()
+ properties = properties.withOrderSpec(new OrderSpec(requiredOrderKeys));
+ }
+
+ RequireProperties requireProperties = RequireProperties.of(properties);
+ return physicalWindow.withRequirePropertiesAndChild(requireProperties, root);
+ }
+
+ /* ********************************************************************************************
+ * WindowFunctionRelatedGroups
+ * ******************************************************************************************** */
+
+ // todo: can we simplify the following three algorithms?
+ private List<WindowFrameGroup> createCommonWindowFrameGroups(List<NamedExpression> windowList) {
+ List<WindowFrameGroup> windowFrameGroupList = Lists.newArrayList();
+ for (int i = 0; i < windowList.size(); i++) {
+ NamedExpression windowAlias = windowList.get(i);
+
+ boolean matched = false;
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ if (windowFrameGroup.isCompatible(windowAlias)) {
+ windowFrameGroup.addGroupMember(windowAlias);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ windowFrameGroupList.add(new WindowFrameGroup(windowAlias));
+ }
+ }
+
+ for (WindowFrameGroup wfg : windowFrameGroupList) {
+ wfg.setTupleSize(wfg.groupList.stream().mapToInt(window -> window.child(0).getDataType().width()).sum());
+ }
+
+ return windowFrameGroupList;
+ }
+
+ private List<OrderKeyGroup> createCommonOrderKeyGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ List<OrderKeyGroup> orderKeyGroupList = Lists.newArrayList();
+
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ boolean matched = false;
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ if (orderKeyGroup.isCompatible(windowFrameGroup)) {
+ orderKeyGroup.addGroupMember(windowFrameGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ orderKeyGroupList.add(new OrderKeyGroup(windowFrameGroup));
+ }
+ }
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ okg.setTupleSize(okg.getGroupList().stream().mapToInt(WindowFrameGroup::getTupleSize).sum());
+ }
+
+ return orderKeyGroupList;
+ }
+
+ private List<PartitionKeyGroup> createCommonPartitionKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ List<PartitionKeyGroup> partitionKeyGroupList = Lists.newArrayList();
+
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ boolean matched = false;
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ if (partitionKeyGroup.isCompatible(orderKeyGroup)) {
+ partitionKeyGroup.addGroupMember(orderKeyGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ partitionKeyGroupList.add(new PartitionKeyGroup(orderKeyGroup));
+ }
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ pkg.setTupleSize(pkg.getGroupList().stream().mapToInt(OrderKeyGroup::getTupleSize).sum());
+ }
+
+ return partitionKeyGroupList;
+ }
+
+ private void mergeOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (OrderKeyGroup okg1 : orderKeyGroupList) {
+ for (OrderKeyGroup okg2 : orderKeyGroupList) {
+ if (okg1 != okg2 && okg2.isPrefixOf(okg1)) {
+ // okg2 ∈ okg1
+ okg1.absorb(okg2);
+ orderKeyGroupList.remove(okg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ private void mergePartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (PartitionKeyGroup pkg1 : partitionKeyGroupList) {
+ for (PartitionKeyGroup pkg2 : partitionKeyGroupList) {
+ if (pkg1 != pkg2) {
+ // todo:
+ pkg1.absorb(pkg2);
+ partitionKeyGroupList.remove(pkg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
Review Comment:
should we need to update tuple size after merge?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
+ // todo: partitionByEq and orderByEq?
+ // requiredProperties:
+ // Distribution: partitionKeys
+ // Order: requiredOrderKeys
+ PhysicalWindow physicalWindow = new PhysicalWindow<>(
+ logicalWindow.getOutputExpressions(),
+ logicalWindow.getWindowExpressions(),
+ windowFrameGroup,
+ logicalWindow.getLogicalProperties(),
+ root);
+ // todo: add isAnalyticSort to physicalWindow
+ if (windowFrameGroup.partitionKeyList.isEmpty() && requiredOrderKeys.isEmpty()) {
+ return physicalWindow.withRequirePropertiesAndChild(RequireProperties.followParent(), root);
+ }
+
+ PhysicalProperties properties;
+ if (windowFrameGroup.partitionKeyList.isEmpty()) {
+ properties = new PhysicalProperties(new OrderSpec(requiredOrderKeys));
+ } else {
+ // todo: add new ShuffleType for window, like ShuffleType.WINDOW
+ properties = PhysicalProperties.createHash(
+ windowFrameGroup.partitionKeyList, DistributionSpecHash.ShuffleType.ENFORCED);
+ // requiredOrderKeys contain partitionKeys, so there is no need to check if requiredOrderKeys.isEmpty()
+ properties = properties.withOrderSpec(new OrderSpec(requiredOrderKeys));
+ }
+
+ RequireProperties requireProperties = RequireProperties.of(properties);
+ return physicalWindow.withRequirePropertiesAndChild(requireProperties, root);
+ }
+
+ /* ********************************************************************************************
+ * WindowFunctionRelatedGroups
+ * ******************************************************************************************** */
+
+ // todo: can we simplify the following three algorithms?
+ private List<WindowFrameGroup> createCommonWindowFrameGroups(List<NamedExpression> windowList) {
+ List<WindowFrameGroup> windowFrameGroupList = Lists.newArrayList();
+ for (int i = 0; i < windowList.size(); i++) {
+ NamedExpression windowAlias = windowList.get(i);
+
+ boolean matched = false;
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ if (windowFrameGroup.isCompatible(windowAlias)) {
+ windowFrameGroup.addGroupMember(windowAlias);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ windowFrameGroupList.add(new WindowFrameGroup(windowAlias));
+ }
+ }
+
+ for (WindowFrameGroup wfg : windowFrameGroupList) {
+ wfg.setTupleSize(wfg.groupList.stream().mapToInt(window -> window.child(0).getDataType().width()).sum());
+ }
+
+ return windowFrameGroupList;
+ }
+
+ private List<OrderKeyGroup> createCommonOrderKeyGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ List<OrderKeyGroup> orderKeyGroupList = Lists.newArrayList();
+
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ boolean matched = false;
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ if (orderKeyGroup.isCompatible(windowFrameGroup)) {
+ orderKeyGroup.addGroupMember(windowFrameGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ orderKeyGroupList.add(new OrderKeyGroup(windowFrameGroup));
+ }
+ }
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ okg.setTupleSize(okg.getGroupList().stream().mapToInt(WindowFrameGroup::getTupleSize).sum());
+ }
+
+ return orderKeyGroupList;
+ }
+
+ private List<PartitionKeyGroup> createCommonPartitionKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ List<PartitionKeyGroup> partitionKeyGroupList = Lists.newArrayList();
+
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ boolean matched = false;
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ if (partitionKeyGroup.isCompatible(orderKeyGroup)) {
+ partitionKeyGroup.addGroupMember(orderKeyGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ partitionKeyGroupList.add(new PartitionKeyGroup(orderKeyGroup));
+ }
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ pkg.setTupleSize(pkg.getGroupList().stream().mapToInt(OrderKeyGroup::getTupleSize).sum());
+ }
+
+ return partitionKeyGroupList;
+ }
+
+ private void mergeOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (OrderKeyGroup okg1 : orderKeyGroupList) {
+ for (OrderKeyGroup okg2 : orderKeyGroupList) {
+ if (okg1 != okg2 && okg2.isPrefixOf(okg1)) {
+ // okg2 ∈ okg1
+ okg1.absorb(okg2);
+ orderKeyGroupList.remove(okg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ private void mergePartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (PartitionKeyGroup pkg1 : partitionKeyGroupList) {
+ for (PartitionKeyGroup pkg2 : partitionKeyGroupList) {
+ if (pkg1 != pkg2) {
+ // todo:
+ pkg1.absorb(pkg2);
+ partitionKeyGroupList.remove(pkg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ /**
+ * [This comment references AnalyticPlanner.orderGroups()]
+ *
+ * Order partition groups (and the SortGroups, WindowFrameGroups within them) by increasing
+ * totalOutputTupleSize. This minimizes the total volume of data that needs to be
+ * repartitioned and sorted.
+ *
+ * Always move the non-partitioning partition group to the end, if it exists.
+ */
+ private void sortPartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+
+ PartitionKeyGroup noPartition = null;
+ // after createCommonPartitionKeyGroups(), there will be at most one empty partition.
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ if (pkg.partitionKeyList.isEmpty()) {
+ noPartition = pkg;
+ partitionKeyGroupList.remove(noPartition);
+ break;
+ }
+ }
+
+ if (noPartition != null) {
+ partitionKeyGroupList.remove(noPartition);
+ }
+
+ partitionKeyGroupList.sort((pkg1, pkg2) ->
+ Integer.compare(pkg1.getTupleSize() - pkg2.getTupleSize(), 0)
+ );
+
+ if (noPartition != null) {
+ partitionKeyGroupList.add(noPartition);
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ sortOrderKeyGroups(pkg.getGroupList());
+ }
+
+ }
+
+ private void sortOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ orderKeyGroupList.sort((okg1, okg2) ->
+ Integer.compare(okg1.getTupleSize() - okg2.getTupleSize(), 0)
+ );
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ sortWindowFrameGroups(okg.getGroupList());
+ }
+ }
+
+ private void sortWindowFrameGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ windowFrameGroupList.sort((wfg1, wfg2) ->
+ Integer.compare(wfg1.getTupleSize() - wfg2.getTupleSize(), 0)
+ );
+ }
+
+ /**
+ * Window Functions that have common PartitionKeys, OrderKeys and WindowFrame
+ */
+ public static class WindowFrameGroup extends WindowFunctionRelatedGroup<NamedExpression> {
+
+ private final List<Expression> partitionKeyList;
+ private final List<OrderKey> orderKeyList;
+ private final WindowFrame windowFrame;
+
+ public WindowFrameGroup(NamedExpression windowAlias) {
+ Window window = (Window) (windowAlias.child(0));
+ partitionKeyList = window.getPartitionKeyList().orElse(Lists.newArrayList());
+ orderKeyList = window.getOrderKeyList().orElse(Lists.newArrayList());
+ windowFrame = window.getWindowFrame().orElse(null);
+ groupList.add(windowAlias);
+ }
+
+ @Override
+ public void addGroupMember(NamedExpression windowAlias) {
+ groupList.add(windowAlias);
+ }
+
+ @Override
+ public boolean isCompatible(NamedExpression windowAlias) {
+ // The comparison of PartitionKey is irrelevant to key's order,
+ // but not in OrderKey' comparison.
+ Window window = (Window) (windowAlias.child(0));
+
+ List<Expression> otherPartitionKeyList = window.getPartitionKeyList().orElse(null);
+ List<OrderKey> otherOrderKeyList = window.getOrderKeyList().orElse(null);
+ WindowFrame otherWindowFrame = window.getWindowFrame().orElse(null);
+
+ // for PartitionKeys, we don't care about the order of each key, so we use isEqualCollection() to compare
+ // whether these two lists have same keys; but for OrderKeys, the order of each key also make sense, so
+ // we use equals() to compare both the elements and their order in these two lists.
+ if (CollectionUtils.isEqualCollection(partitionKeyList, otherPartitionKeyList)
+ && orderKeyList.equals(otherOrderKeyList)) {
+ // CollectionUtils.isEqualCollection() is absolutely equals to Expr.equalSets()
+ if ((windowFrame == null && otherWindowFrame == null) || windowFrame.equals(otherWindowFrame)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder sb = new StringBuilder();
+ sb.append("PartitionKeys: ").append(partitionKeyList.stream()
+ .map(Expression::toString)
+ .collect(Collectors.joining(", ", "", ", ")));
+
+ sb.append("OrderKeys: ").append(orderKeyList.stream()
+ .map(OrderKey::toString)
+ .collect(Collectors.joining(", ", "", ", ")));
+
+ sb.append("WindowFrame: ").append(windowFrame);
+ return sb.toString();
+ }
+
+ public List<Expression> getPartitionKeyList() {
+ return partitionKeyList;
+ }
+
+ public List<OrderKey> getOrderKeyList() {
+ return orderKeyList;
+ }
+
+ public WindowFrame getWindowFrame() {
+ return windowFrame;
+ }
+
+ }
+
+ /**
+ * Window Functions that have common PartitionKeys and OrderKeys.
+ */
+ private static class OrderKeyGroup extends WindowFunctionRelatedGroup<WindowFrameGroup> {
+
+ private final List<Expression> partitionKeyList;
+ private final List<OrderKey> orderKeyList;
+
+ public OrderKeyGroup(WindowFrameGroup windowFrameGroup) {
+ partitionKeyList = windowFrameGroup.partitionKeyList;
+ orderKeyList = windowFrameGroup.orderKeyList;
+ groupList.add(windowFrameGroup);
+ }
+
+ @Override
+ public boolean isCompatible(WindowFrameGroup windowFrameGroup) {
+ return CollectionUtils.isEqualCollection(partitionKeyList, windowFrameGroup.partitionKeyList)
+ && orderKeyList.equals(windowFrameGroup.orderKeyList);
+ }
+
+ /**
+ * check if this okg isPrefixOf other, which means that other okg can cover this
+ * e.g. (a, b) is a prefix of (a, b, c), but is NOT a prefix of (b, a, c) or (b, c)
+ */
+ public boolean isPrefixOf(OrderKeyGroup otherOkg) {
+ if (orderKeyList.size() > otherOkg.orderKeyList.size()) {
+ return false;
+ }
+ if (!CollectionUtils.isEqualCollection(partitionKeyList, otherOkg.partitionKeyList)) {
+ return false;
+ }
+
+ for (int i = 0; i < orderKeyList.size(); i++) {
+ if (!orderKeyList.get(i).equals(otherOkg.orderKeyList.get(i))) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * add all of otherOkg's WindowFrameGroups to this groupList
+ */
+ public void absorb(OrderKeyGroup otherOkg) {
+ groupList.addAll(otherOkg.groupList);
+ }
+
+ }
+
+ /**
+ * Window Functions that have common PartitionKeys.
+ */
+ private static class PartitionKeyGroup extends WindowFunctionRelatedGroup<OrderKeyGroup> {
+ public List<Expression> partitionKeyList;
+
+ public PartitionKeyGroup(OrderKeyGroup orderKeyGroup) {
+ partitionKeyList = orderKeyGroup.partitionKeyList;
+ groupList.add(orderKeyGroup);
+ }
+
+ @Override
+ public boolean isCompatible(OrderKeyGroup orderKeyGroup) {
+ return CollectionUtils.isEqualCollection(partitionKeyList, orderKeyGroup.partitionKeyList);
+ }
+
+ /**
+ * absorb other into this:
+ * - partitionKeyList will be the intersection of this two group
+ * - groupList(orderKeyGroup) will be the union of this two group
+ */
+ public void absorb(PartitionKeyGroup otherPkg) {
+ List<Expression> intersectPartitionKeys = partitionKeyList.stream()
+ .filter(expression -> otherPkg.partitionKeyList.contains(expression))
+ .collect(Collectors.toList());
+ partitionKeyList = intersectPartitionKeys;
Review Comment:
```suggestion
partitionKeyList = partitionKeyList.stream()
.filter(expression -> otherPkg.partitionKeyList.contains(expression))
.collect(Collectors.toList());
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
+ // todo: partitionByEq and orderByEq?
+ // requiredProperties:
+ // Distribution: partitionKeys
+ // Order: requiredOrderKeys
+ PhysicalWindow physicalWindow = new PhysicalWindow<>(
+ logicalWindow.getOutputExpressions(),
+ logicalWindow.getWindowExpressions(),
+ windowFrameGroup,
+ logicalWindow.getLogicalProperties(),
+ root);
+ // todo: add isAnalyticSort to physicalWindow
+ if (windowFrameGroup.partitionKeyList.isEmpty() && requiredOrderKeys.isEmpty()) {
+ return physicalWindow.withRequirePropertiesAndChild(RequireProperties.followParent(), root);
+ }
+
+ PhysicalProperties properties;
+ if (windowFrameGroup.partitionKeyList.isEmpty()) {
+ properties = new PhysicalProperties(new OrderSpec(requiredOrderKeys));
+ } else {
+ // todo: add new ShuffleType for window, like ShuffleType.WINDOW
+ properties = PhysicalProperties.createHash(
+ windowFrameGroup.partitionKeyList, DistributionSpecHash.ShuffleType.ENFORCED);
+ // requiredOrderKeys contain partitionKeys, so there is no need to check if requiredOrderKeys.isEmpty()
+ properties = properties.withOrderSpec(new OrderSpec(requiredOrderKeys));
+ }
+
+ RequireProperties requireProperties = RequireProperties.of(properties);
+ return physicalWindow.withRequirePropertiesAndChild(requireProperties, root);
+ }
+
+ /* ********************************************************************************************
+ * WindowFunctionRelatedGroups
+ * ******************************************************************************************** */
+
+ // todo: can we simplify the following three algorithms?
+ private List<WindowFrameGroup> createCommonWindowFrameGroups(List<NamedExpression> windowList) {
+ List<WindowFrameGroup> windowFrameGroupList = Lists.newArrayList();
+ for (int i = 0; i < windowList.size(); i++) {
+ NamedExpression windowAlias = windowList.get(i);
+
+ boolean matched = false;
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ if (windowFrameGroup.isCompatible(windowAlias)) {
+ windowFrameGroup.addGroupMember(windowAlias);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ windowFrameGroupList.add(new WindowFrameGroup(windowAlias));
+ }
+ }
+
+ for (WindowFrameGroup wfg : windowFrameGroupList) {
+ wfg.setTupleSize(wfg.groupList.stream().mapToInt(window -> window.child(0).getDataType().width()).sum());
+ }
+
+ return windowFrameGroupList;
+ }
+
+ private List<OrderKeyGroup> createCommonOrderKeyGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ List<OrderKeyGroup> orderKeyGroupList = Lists.newArrayList();
+
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ boolean matched = false;
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ if (orderKeyGroup.isCompatible(windowFrameGroup)) {
+ orderKeyGroup.addGroupMember(windowFrameGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ orderKeyGroupList.add(new OrderKeyGroup(windowFrameGroup));
+ }
+ }
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ okg.setTupleSize(okg.getGroupList().stream().mapToInt(WindowFrameGroup::getTupleSize).sum());
+ }
+
+ return orderKeyGroupList;
+ }
+
+ private List<PartitionKeyGroup> createCommonPartitionKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ List<PartitionKeyGroup> partitionKeyGroupList = Lists.newArrayList();
+
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ boolean matched = false;
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ if (partitionKeyGroup.isCompatible(orderKeyGroup)) {
+ partitionKeyGroup.addGroupMember(orderKeyGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ partitionKeyGroupList.add(new PartitionKeyGroup(orderKeyGroup));
+ }
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ pkg.setTupleSize(pkg.getGroupList().stream().mapToInt(OrderKeyGroup::getTupleSize).sum());
+ }
+
+ return partitionKeyGroupList;
+ }
+
+ private void mergeOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (OrderKeyGroup okg1 : orderKeyGroupList) {
+ for (OrderKeyGroup okg2 : orderKeyGroupList) {
+ if (okg1 != okg2 && okg2.isPrefixOf(okg1)) {
+ // okg2 ∈ okg1
+ okg1.absorb(okg2);
+ orderKeyGroupList.remove(okg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ private void mergePartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (PartitionKeyGroup pkg1 : partitionKeyGroupList) {
+ for (PartitionKeyGroup pkg2 : partitionKeyGroupList) {
+ if (pkg1 != pkg2) {
+ // todo:
Review Comment:
remove this todo
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
Review Comment:
```suggestion
private PhysicalWindow<Plan> createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
LogicalWindow<Plan> logicalWindow, List<OrderKey> requiredOrderKeys) {
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
+ // todo: partitionByEq and orderByEq?
+ // requiredProperties:
+ // Distribution: partitionKeys
+ // Order: requiredOrderKeys
+ PhysicalWindow physicalWindow = new PhysicalWindow<>(
Review Comment:
```suggestion
PhysicalWindow<Plan> physicalWindow = new PhysicalWindow<>(
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
+ // todo: partitionByEq and orderByEq?
+ // requiredProperties:
+ // Distribution: partitionKeys
+ // Order: requiredOrderKeys
+ PhysicalWindow physicalWindow = new PhysicalWindow<>(
+ logicalWindow.getOutputExpressions(),
+ logicalWindow.getWindowExpressions(),
+ windowFrameGroup,
+ logicalWindow.getLogicalProperties(),
+ root);
+ // todo: add isAnalyticSort to physicalWindow
+ if (windowFrameGroup.partitionKeyList.isEmpty() && requiredOrderKeys.isEmpty()) {
+ return physicalWindow.withRequirePropertiesAndChild(RequireProperties.followParent(), root);
+ }
+
+ PhysicalProperties properties;
+ if (windowFrameGroup.partitionKeyList.isEmpty()) {
+ properties = new PhysicalProperties(new OrderSpec(requiredOrderKeys));
+ } else {
+ // todo: add new ShuffleType for window, like ShuffleType.WINDOW
+ properties = PhysicalProperties.createHash(
+ windowFrameGroup.partitionKeyList, DistributionSpecHash.ShuffleType.ENFORCED);
+ // requiredOrderKeys contain partitionKeys, so there is no need to check if requiredOrderKeys.isEmpty()
+ properties = properties.withOrderSpec(new OrderSpec(requiredOrderKeys));
Review Comment:
i don't think we should set orderSpec here. Because orderSpec is indicate the order in one `INSTANCE`. but window need order in one `PARTITION`
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalWindow.java:
##########
@@ -0,0 +1,189 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.properties.RequirePropertiesSupplier;
+import org.apache.doris.nereids.rules.implementation.LogicalWindowToPhysicalWindow.WindowFrameGroup;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.Window;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+import org.apache.doris.statistics.StatsDeriveResult;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * physical node for window function
+ */
+public class PhysicalWindow<CHILD_TYPE extends Plan> extends PhysicalUnary<CHILD_TYPE> implements Window,
+ RequirePropertiesSupplier<PhysicalWindow<CHILD_TYPE>> {
+
+ private List<NamedExpression> outputExpressions;
+ private List<NamedExpression> windowExpressions;
+ private WindowFrameGroup windowFrameGroup;
+ private RequireProperties requireProperties;
+
+ public PhysicalWindow(List<NamedExpression> outputExpressions, List<NamedExpression> windowExpressions,
+ WindowFrameGroup windowFrameGroup, LogicalProperties logicalProperties, CHILD_TYPE child) {
+ this(outputExpressions, windowExpressions, windowFrameGroup, null, Optional.empty(), logicalProperties, child);
+ }
+
+ /** constructor for PhysicalWindow */
+ public PhysicalWindow(List<NamedExpression> outputExpressions, List<NamedExpression> windowExpressions,
+ WindowFrameGroup windowFrameGroup, RequireProperties requireProperties,
+ Optional<GroupExpression> groupExpression, LogicalProperties logicalProperties,
+ CHILD_TYPE child) {
+ super(PlanType.PHYSICAL_WINDOW, groupExpression, logicalProperties, child);
+ this.outputExpressions = outputExpressions;
+ this.windowExpressions = windowExpressions;
Review Comment:
ImmutableList.copyOf and Objects.requireNotNull
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalWindow.java:
##########
@@ -0,0 +1,189 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.properties.RequirePropertiesSupplier;
+import org.apache.doris.nereids.rules.implementation.LogicalWindowToPhysicalWindow.WindowFrameGroup;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.Window;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+import org.apache.doris.statistics.StatsDeriveResult;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * physical node for window function
+ */
+public class PhysicalWindow<CHILD_TYPE extends Plan> extends PhysicalUnary<CHILD_TYPE> implements Window,
+ RequirePropertiesSupplier<PhysicalWindow<CHILD_TYPE>> {
+
+ private List<NamedExpression> outputExpressions;
+ private List<NamedExpression> windowExpressions;
+ private WindowFrameGroup windowFrameGroup;
+ private RequireProperties requireProperties;
Review Comment:
final
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
+ // todo: partitionByEq and orderByEq?
+ // requiredProperties:
+ // Distribution: partitionKeys
+ // Order: requiredOrderKeys
+ PhysicalWindow physicalWindow = new PhysicalWindow<>(
+ logicalWindow.getOutputExpressions(),
+ logicalWindow.getWindowExpressions(),
+ windowFrameGroup,
+ logicalWindow.getLogicalProperties(),
+ root);
+ // todo: add isAnalyticSort to physicalWindow
+ if (windowFrameGroup.partitionKeyList.isEmpty() && requiredOrderKeys.isEmpty()) {
+ return physicalWindow.withRequirePropertiesAndChild(RequireProperties.followParent(), root);
+ }
+
+ PhysicalProperties properties;
+ if (windowFrameGroup.partitionKeyList.isEmpty()) {
+ properties = new PhysicalProperties(new OrderSpec(requiredOrderKeys));
+ } else {
+ // todo: add new ShuffleType for window, like ShuffleType.WINDOW
+ properties = PhysicalProperties.createHash(
+ windowFrameGroup.partitionKeyList, DistributionSpecHash.ShuffleType.ENFORCED);
+ // requiredOrderKeys contain partitionKeys, so there is no need to check if requiredOrderKeys.isEmpty()
+ properties = properties.withOrderSpec(new OrderSpec(requiredOrderKeys));
+ }
+
+ RequireProperties requireProperties = RequireProperties.of(properties);
+ return physicalWindow.withRequirePropertiesAndChild(requireProperties, root);
+ }
+
+ /* ********************************************************************************************
+ * WindowFunctionRelatedGroups
+ * ******************************************************************************************** */
+
+ // todo: can we simplify the following three algorithms?
+ private List<WindowFrameGroup> createCommonWindowFrameGroups(List<NamedExpression> windowList) {
+ List<WindowFrameGroup> windowFrameGroupList = Lists.newArrayList();
+ for (int i = 0; i < windowList.size(); i++) {
+ NamedExpression windowAlias = windowList.get(i);
+
+ boolean matched = false;
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ if (windowFrameGroup.isCompatible(windowAlias)) {
+ windowFrameGroup.addGroupMember(windowAlias);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ windowFrameGroupList.add(new WindowFrameGroup(windowAlias));
+ }
+ }
+
+ for (WindowFrameGroup wfg : windowFrameGroupList) {
+ wfg.setTupleSize(wfg.groupList.stream().mapToInt(window -> window.child(0).getDataType().width()).sum());
+ }
+
+ return windowFrameGroupList;
+ }
+
+ private List<OrderKeyGroup> createCommonOrderKeyGroups(List<WindowFrameGroup> windowFrameGroupList) {
+ List<OrderKeyGroup> orderKeyGroupList = Lists.newArrayList();
+
+ for (WindowFrameGroup windowFrameGroup : windowFrameGroupList) {
+ boolean matched = false;
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ if (orderKeyGroup.isCompatible(windowFrameGroup)) {
+ orderKeyGroup.addGroupMember(windowFrameGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ orderKeyGroupList.add(new OrderKeyGroup(windowFrameGroup));
+ }
+ }
+
+ for (OrderKeyGroup okg : orderKeyGroupList) {
+ okg.setTupleSize(okg.getGroupList().stream().mapToInt(WindowFrameGroup::getTupleSize).sum());
+ }
+
+ return orderKeyGroupList;
+ }
+
+ private List<PartitionKeyGroup> createCommonPartitionKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ List<PartitionKeyGroup> partitionKeyGroupList = Lists.newArrayList();
+
+ for (OrderKeyGroup orderKeyGroup : orderKeyGroupList) {
+ boolean matched = false;
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ if (partitionKeyGroup.isCompatible(orderKeyGroup)) {
+ partitionKeyGroup.addGroupMember(orderKeyGroup);
+ matched = true;
+ break;
+ }
+ }
+ if (!matched) {
+ partitionKeyGroupList.add(new PartitionKeyGroup(orderKeyGroup));
+ }
+ }
+
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ pkg.setTupleSize(pkg.getGroupList().stream().mapToInt(OrderKeyGroup::getTupleSize).sum());
+ }
+
+ return partitionKeyGroupList;
+ }
+
+ private void mergeOrderKeyGroups(List<OrderKeyGroup> orderKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (OrderKeyGroup okg1 : orderKeyGroupList) {
+ for (OrderKeyGroup okg2 : orderKeyGroupList) {
+ if (okg1 != okg2 && okg2.isPrefixOf(okg1)) {
+ // okg2 ∈ okg1
+ okg1.absorb(okg2);
+ orderKeyGroupList.remove(okg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ private void mergePartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+ boolean merged = true;
+
+ while (merged) {
+ merged = false;
+ for (PartitionKeyGroup pkg1 : partitionKeyGroupList) {
+ for (PartitionKeyGroup pkg2 : partitionKeyGroupList) {
+ if (pkg1 != pkg2) {
+ // todo:
+ pkg1.absorb(pkg2);
+ partitionKeyGroupList.remove(pkg2);
+ merged = true;
+ break;
+ }
+ }
+ if (merged) {
+ break;
+ }
+ }
+ }
+ }
+
+ /**
+ * [This comment references AnalyticPlanner.orderGroups()]
+ *
+ * Order partition groups (and the SortGroups, WindowFrameGroups within them) by increasing
+ * totalOutputTupleSize. This minimizes the total volume of data that needs to be
+ * repartitioned and sorted.
+ *
+ * Always move the non-partitioning partition group to the end, if it exists.
+ */
+ private void sortPartitionKeyGroups(List<PartitionKeyGroup> partitionKeyGroupList) {
+
+ PartitionKeyGroup noPartition = null;
+ // after createCommonPartitionKeyGroups(), there will be at most one empty partition.
+ for (PartitionKeyGroup pkg : partitionKeyGroupList) {
+ if (pkg.partitionKeyList.isEmpty()) {
+ noPartition = pkg;
+ partitionKeyGroupList.remove(noPartition);
+ break;
+ }
+ }
+
+ if (noPartition != null) {
+ partitionKeyGroupList.remove(noPartition);
+ }
Review Comment:
remove twice
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##########
@@ -0,0 +1,539 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.implementation;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.properties.OrderSpec;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.properties.RequireProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.logical.CheckAndStandardizeWindowFunctionAndFrame;
+import org.apache.doris.nereids.rules.rewrite.logical.ExtractWindowExpression;
+import org.apache.doris.nereids.rules.rewrite.logical.NormalizeWindow;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Window;
+import org.apache.doris.nereids.trees.expressions.WindowFrame;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalWindow;
+
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Implementation rule that convert logical window to physical window, and add RequiredProperties
+ *
+ * step 1: compute three kinds of group:
+ * WindowFrameGroup: maintain windows with same PartitionKey, OrderKey and WindowFrame
+ * OrderKeyGroup: maintain windows with same PartitionKey and OrderKey
+ * PartitionKeyGroup: maintain windows with same PartitionKey
+ * step 2: sort PartitionKeyGroup with increasing order of tupleSize
+ * step 3: for every WindowFrameGroup of each SortGroup, generate one PhysicalWindow node, with common PartitionKeys,
+ * OrderKeys, unique WindowFrame and a function list.
+ * step 4: for each PhysicalWindow, generate RequiredProperties, including PartitionKey for DistributionSpec,
+ * and (PartitionKey + OrderKey) for OrderSpec.
+ */
+@DependsRules({
+ NormalizeWindow.class,
+ CheckAndStandardizeWindowFunctionAndFrame.class,
+ ExtractWindowExpression.class
+})
+public class LogicalWindowToPhysicalWindow extends OneImplementationRuleFactory {
+
+ @Override
+ public Rule build() {
+
+ return RuleType.LOGICAL_WINDOW_TO_PHYSICAL_WINDOW_RULE.build(
+ logicalWindow().when(LogicalWindow::isNormalized)
+ .when(LogicalWindow::isChecked)
+ .thenApply(ctx -> resolveWindow(ctx.cascadesContext, ctx.root))
+ );
+ }
+
+ /**
+ * main procedure
+ */
+ private PhysicalWindow resolveWindow(CascadesContext ctx, LogicalWindow<GroupPlan> logicalWindow) {
+ // todo: remove windowExpressions from LogicalWindow and rule ExtractWindowExpressions.
+ // Only add this variable in PhysicalWindow
+ List<NamedExpression> windowList = logicalWindow.getWindowExpressions();
+
+ /////////// create three kinds of groups and compute tupleSize of each
+ // windowFrameGroup
+ List<WindowFrameGroup> windowFrameGroupList = createCommonWindowFrameGroups(windowList);
+
+ // orderKeyGroup
+ List<OrderKeyGroup> orderKeyGroupList = createCommonOrderKeyGroups(windowFrameGroupList);
+ mergeOrderKeyGroups(orderKeyGroupList);
+
+ // partitionKeyGroup
+ List<PartitionKeyGroup> partitionKeyGroupList = createCommonPartitionKeyGroups(orderKeyGroupList);
+ mergePartitionKeyGroups(partitionKeyGroupList);
+
+ // sort groups
+ sortPartitionKeyGroups(partitionKeyGroupList);
+
+ // eliminate LogicalWindow, and replaced it with new LogicalWindow and LogicalSort
+ Plan newRoot = logicalWindow.child();
+ for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
+ for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groupList) {
+ // create LogicalSort for each OrderKeyGroup;
+ // in OrderKeyGroup, create LogicalWindow for each WindowFrameGroup
+ newRoot = createLogicalPlanNodeForWindowFunctions(newRoot, orderKeyGroup, logicalWindow, ctx);
+ }
+ }
+ return (PhysicalWindow) newRoot;
+ }
+
+ /* ********************************************************************************************
+ * create PhysicalWindow and PhysicalSort
+ * ******************************************************************************************** */
+
+ private Plan createLogicalPlanNodeForWindowFunctions(Plan root, OrderKeyGroup orderKeyGroup,
+ LogicalWindow logicalWindow, CascadesContext ctx) {
+ // PhysicalSort node for orderKeys; if there exists no orderKey, newRoot = root
+ // Plan newRoot = createPhysicalSortNode(root, orderKeyGroup, ctx);
+ Plan newRoot = root;
+
+ // we will not add PhysicalSort in this step, but generate it if necessary with the ability of enforcer by
+ // setting RequiredProperties
+ List<OrderKey> requiredOrderKeys = generateKeysNeedToBeSorted(orderKeyGroup);
+
+ // PhysicalWindow nodes for each different window frame, so at least one PhysicalWindow node will be added
+ for (WindowFrameGroup windowFrameGroup : orderKeyGroup.groupList) {
+ newRoot = createPhysicalWindow(newRoot, windowFrameGroup, logicalWindow, requiredOrderKeys);
+ }
+
+ return newRoot;
+ }
+
+ private List<OrderKey> generateKeysNeedToBeSorted(OrderKeyGroup orderKeyGroup) {
+ // all keys that need to be sorted, which includes BOTH partitionKeys and orderKeys from this group
+ List<OrderKey> keysNeedToBeSortedList = Lists.newArrayList();
+
+ // used as SortNode.isAnalyticSort, but it is unnecessary to add it in LogicalSort
+ if (!orderKeyGroup.partitionKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.partitionKeyList.stream().map(partitionKey -> {
+ // todo: haven't support isNullFirst, and its default value is false(see AnalyticPlanner#line403,
+ // but in LogicalPlanBuilder, its default value is true)
+ return new OrderKey(partitionKey, true, false);
+ }).collect(Collectors.toList()));
+ }
+
+ if (!orderKeyGroup.orderKeyList.isEmpty()) {
+ keysNeedToBeSortedList.addAll(orderKeyGroup.orderKeyList);
+ }
+ return keysNeedToBeSortedList;
+ }
+
+ private PhysicalWindow createPhysicalWindow(Plan root, WindowFrameGroup windowFrameGroup,
+ LogicalWindow logicalWindow, List<OrderKey> requiredOrderKeys) {
+ // todo: partitionByEq and orderByEq?
+ // requiredProperties:
+ // Distribution: partitionKeys
+ // Order: requiredOrderKeys
+ PhysicalWindow physicalWindow = new PhysicalWindow<>(
+ logicalWindow.getOutputExpressions(),
+ logicalWindow.getWindowExpressions(),
+ windowFrameGroup,
+ logicalWindow.getLogicalProperties(),
+ root);
+ // todo: add isAnalyticSort to physicalWindow
+ if (windowFrameGroup.partitionKeyList.isEmpty() && requiredOrderKeys.isEmpty()) {
+ return physicalWindow.withRequirePropertiesAndChild(RequireProperties.followParent(), root);
+ }
+
+ PhysicalProperties properties;
+ if (windowFrameGroup.partitionKeyList.isEmpty()) {
+ properties = new PhysicalProperties(new OrderSpec(requiredOrderKeys));
+ } else {
+ // todo: add new ShuffleType for window, like ShuffleType.WINDOW
Review Comment:
what is the purpose of ShuffleType.WINDOW?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1092848340
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -945,15 +954,99 @@ public Expression visitFunctionCall(DorisParser.FunctionCallContext ctx) {
if (!unboundStars.get(0).getQualifier().isEmpty()) {
throw new ParseException("'*' can not has qualifier: " + unboundStars.size(), ctx);
}
+ if (ctx.windowSpec() != null) {
+ throw new ParseException(
+ "COUNT(*) isn't supported as window function; can use COUNT(col)", ctx);
+ }
return new Count();
}
throw new ParseException("'*' can only be used in conjunction with COUNT: " + functionName, ctx);
} else {
- return new UnboundFunction(functionName, isDistinct, params);
+ UnboundFunction function = new UnboundFunction(functionName, isDistinct, params);
+ if (ctx.windowSpec() != null) {
+ if (isDistinct) {
+ throw new ParseException("DISTINCT not allowed in window function: " + functionName, ctx);
+ }
+ return withWindowSpec(ctx.windowSpec(), function);
+ }
+ return function;
}
});
}
+ /**
+ * deal with window function definition
+ */
+ private Window withWindowSpec(WindowSpecContext ctx, Expression function) {
+ Optional<List<Expression>> partitionKeys = optionalVisit(ctx.partitionClause(),
+ () -> visit(ctx.partitionClause().expression(), Expression.class));
+
+ Optional<List<OrderKey>> orderKeys = optionalVisit(ctx.sortClause(),
+ () -> visit(ctx.sortClause().sortItem(), OrderKey.class));
+
+ Optional<WindowFrame> windowFrame = optionalVisit(ctx.windowFrame(),
+ () -> withWindowFrame(ctx.windowFrame()));
+
+ return new Window(function, partitionKeys, orderKeys, windowFrame);
+ }
+
+ /**
+ * deal with optional expressions
+ */
+ private <T, C> Optional optionalVisit(T ctx, Supplier<C> func) {
+ return Optional.ofNullable(ctx).map(a -> Optional.of(func.get()))
+ .orElse(Optional.empty());
+ }
Review Comment:
```suggestion
private <T, C> Optional<C> optionalVisit(T ctx, Supplier<C> func) {
return Optional.ofNullable(ctx).map(a -> func.get());
}
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -340,13 +350,12 @@ private LogicalPlan withCte(LogicalPlan plan, CteContext ctx) {
public LogicalSubQueryAlias<Plan> visitAliasQuery(AliasQueryContext ctx) {
return ParserUtils.withOrigin(ctx, () -> {
LogicalPlan queryPlan = plan(ctx.query());
- List<String> columnNames = null;
- if (ctx.columnAliases() != null) {
- columnNames = ctx.columnAliases().identifier().stream()
+ Optional<List<String>> columnNames = optionalVisit(ctx.columnAliases(), () ->
+ ctx.columnAliases().identifier().stream()
.map(RuleContext::getText)
- .collect(ImmutableList.toImmutableList());
- }
- return new LogicalSubQueryAlias<>(ctx.identifier().getText(), Optional.ofNullable(columnNames), queryPlan);
+ .collect(ImmutableList.toImmutableList())
+ );
+ return new LogicalSubQueryAlias(ctx.identifier().getText(), columnNames, queryPlan);
Review Comment:
```suggestion
return new LogicalSubQueryAlias<>(ctx.identifier().getText(), columnNames, queryPlan);
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -340,13 +350,12 @@ private LogicalPlan withCte(LogicalPlan plan, CteContext ctx) {
public LogicalSubQueryAlias<Plan> visitAliasQuery(AliasQueryContext ctx) {
return ParserUtils.withOrigin(ctx, () -> {
LogicalPlan queryPlan = plan(ctx.query());
- List<String> columnNames = null;
- if (ctx.columnAliases() != null) {
- columnNames = ctx.columnAliases().identifier().stream()
+ Optional<List<String>> columnNames = optionalVisit(ctx.columnAliases(), () ->
+ ctx.columnAliases().identifier().stream()
Review Comment:
```suggestion
Optional<List<String>> columnNames = optionalVisit(ctx.columnAliases(),
() -> ctx.columnAliases().identifier().stream()
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/FrameUnitsType.java:
##########
@@ -0,0 +1,34 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.window;
+
+/**
+ * frame units types
+ */
+public enum FrameUnitsType {
Review Comment:
since it is only used in WindowFrame, i think it is better to treat it as a inner enum of `WindowFrame`
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -1437,12 +1530,19 @@ private LogicalPlan withProjection(LogicalPlan input, SelectColumnClauseContext
expressions, input, isDistinct);
} else {
List<NamedExpression> projects = getNamedExpressions(selectCtx.namedExpressionSeq());
- return new LogicalProject<>(projects, ImmutableList.of(), input, isDistinct);
+ if (containsWindowExpressions(projects)) {
+ return new LogicalWindow<>(projects, input);
+ }
+ return new LogicalProject<>(projects, Collections.emptyList(), input, isDistinct);
Review Comment:
maybe we should generate LogicalWindow after we do bind.
we chould just treat windowExpression as a scalar expression.
- if we use window function without agg. Then window function will in Projections.
we could generate LogicalProject(LogicalWindow(LogicalProject)) to handle all case.
- if we use window function in agg. Then window function will in Agg's output expressions. After normalize agg it will present in the Project topping on Aggregate. and then we could do the same thing as first scene.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/FrameUnitsType.java:
##########
@@ -0,0 +1,34 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.window;
+
+/**
+ * frame units types
+ */
+public enum FrameUnitsType {
+
+ ROWS("ROWS"),
+ RANGE("RANGE");
+
+ private final String description;
Review Comment:
description is not use anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] YangShaw commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "YangShaw (via GitHub)" <gi...@apache.org>.
YangShaw commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1094411155
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -1437,12 +1530,19 @@ private LogicalPlan withProjection(LogicalPlan input, SelectColumnClauseContext
expressions, input, isDistinct);
} else {
List<NamedExpression> projects = getNamedExpressions(selectCtx.namedExpressionSeq());
- return new LogicalProject<>(projects, ImmutableList.of(), input, isDistinct);
+ if (containsWindowExpressions(projects)) {
+ return new LogicalWindow<>(projects, input);
+ }
+ return new LogicalProject<>(projects, Collections.emptyList(), input, isDistinct);
Review Comment:
nice idea~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1104253370
##########
regression-test/suites/nereids_syntax_p0/window_function.groovy:
##########
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_window_function") {
+ sql "SET enable_nereids_planner=true"
+
+ sql "DROP TABLE IF EXISTS window_test"
+
+ sql """
+ CREATE TABLE `window_test` (
+ `c1` int NULL,
+ `c2` int NULL,
+ `c3` double NULL
+ ) ENGINE=OLAP
+ DISTRIBUTED BY HASH(`c1`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );
+ """
+ sql """INSERT INTO window_test VALUES(1, 1, 1)"""
+ sql """INSERT INTO window_test VALUES(1, 2, 1)"""
+ sql """INSERT INTO window_test VALUES(1, 3, 1)"""
+ sql """INSERT INTO window_test VALUES(2, 1, 1)"""
+ sql """INSERT INTO window_test VALUES(2, 2, 1)"""
+ sql """INSERT INTO window_test VALUES(2, 3, 1)"""
+ sql """INSERT INTO window_test VALUES(1, 1, 2)"""
+ sql """INSERT INTO window_test VALUES(1, 2, 2)"""
+ sql """INSERT INTO window_test VALUES(2, 1, 2)"""
+ sql """INSERT INTO window_test VALUES(2, 2, 2)"""
+ sql """INSERT INTO window_test VALUES(1, 2, null)"""
+ sql """INSERT INTO window_test VALUES(1, 3, null)"""
+ sql """INSERT INTO window_test VALUES(1, null, 3)"""
+ sql """INSERT INTO window_test VALUES(2, null, 3)"""
+
+ sql "SET enable_fallback_to_original_planner=false"
+
+ order_qt_empty_over "SELECT rank() over() FROM window_test"
+ order_qt_with_star "SELECT *, rank() over(partition by c3 order by c2) FROM window_test"
+ order_qt_multi_window_1 "SELECT rank() over(), row_number() over() FROM window_test"
+ order_qt_multi_window_2 """
+ SELECT sum(c1) over(), count(c2) over(), rank() over(partition by c3 order by c2)
+ FROM window_test
+ """
+
+ // todo: fix unknown bug of dense_rank(), ntile() in Nereids
+ // order_qt_select "SELECT dense_rank() over(partition by c3 order by c2) FROM window_test"
+ order_qt_rank "SELECT rank() over(partition by c3 order by c2) FROM window_test"
+ order_qt_row_number "SELECT row_number() over(partition by c3 order by c2) FROM window_test"
+ order_qt_sum "SELECT sum(c1) over(partition by c3 order by c2) FROM window_test"
+ order_qt_avg "SELECT avg(c1) over(partition by c3 order by c2) FROM window_test"
+ order_qt_count "SELECT count(c1) over(partition by c3 order by c2) FROM window_test"
+ order_qt_first_value "SELECT first_value(c1) over(partition by c3 order by c2) FROM window_test"
+ order_qt_last_value "SELECT last_value(c1) over(partition by c3 order by c2) FROM window_test"
+ order_qt_lead "SELECT lead(c1, 1, 111) over(partition by c3 order by c2) FROM window_test"
+ order_qt_lag "SELECT lag(c1, 1, 222) over(partition by c3 order by c2) FROM window_test"
+ // order_qt_lead "SELECT ntile(5) over(partition by c3 order by c2) FROM window_test"
+ order_qt_max "SELECT max(c1) over(partition by c3 order by c2) FROM window_test"
+ order_qt_min "SELECT min(c1) over(partition by c3 order by c2) FROM window_test"
+
+ order_qt_agg_1 """
+ SELECT c1+1, c2, sum(c1), sum(c1) over(partition by c2 order by c1)
+ FROM window_test
+ GROUP BY c1, c2
+ """
+
+ order_qt_agg_2 """
+ SELECT c1, c2, sum(c1+1), sum(c1+1) over(partition by c2 order by c1+1)
+ FROM window_test
+ GROUP BY c1, c2
+ """
+
+ order_qt_agg_3 """
+ SELECT row_number() over(partition by (grouping(c2) + grouping(c3)))
+ FROM window_test
+ GROUP BY ROLLUP (c2, c3)
+ """
+
+ qt_subquery_1 """
+ SELECT *, row_number() over(partition by c1)
+ FROM (
+ SELECT *, row_number() over(partition by c2)
+ FROM window_test
Review Comment:
need order by in over to get a stable result
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] YangShaw commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "YangShaw (via GitHub)" <gi...@apache.org>.
YangShaw commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1094130540
##########
fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4:
##########
@@ -327,6 +332,32 @@ functionIdentifier
| LEFT | RIGHT
;
+windowSpec
+ : name=identifier
+ | LEFT_PAREN name=identifier RIGHT_PAREN
+ | LEFT_PAREN
+ partitionClause?
+ sortClause?
+ windowFrame?
+ RIGHT_PAREN
Review Comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] YangShaw commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "YangShaw (via GitHub)" <gi...@apache.org>.
YangShaw commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1098228130
##########
fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4:
##########
@@ -327,6 +332,32 @@ functionIdentifier
| LEFT | RIGHT
;
+windowSpec
+ : name=identifier
+ | LEFT_PAREN name=identifier RIGHT_PAREN
Review Comment:
haven't yet; I reserve this two line and throw related exceptions during parsing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14397: [feature](nereids)support window function
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14397:
URL: https://github.com/apache/doris/pull/14397#issuecomment-1427905391
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1104276282
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/Ntile.java:
##########
@@ -0,0 +1,82 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.window;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable;
+import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.shape.LeafExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.IntegerType;
+import org.apache.doris.nereids.types.LargeIntType;
+import org.apache.doris.nereids.types.SmallIntType;
+import org.apache.doris.nereids.types.TinyIntType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * window function: Ntile()
+ */
+public class Ntile extends WindowFunction implements LeafExpression, AlwaysNotNullable, ExplicitlyCastableSignature {
+
+ public static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
+ FunctionSignature.ret(BigIntType.INSTANCE).args(TinyIntType.INSTANCE),
+ FunctionSignature.ret(BigIntType.INSTANCE).args(SmallIntType.INSTANCE),
+ FunctionSignature.ret(BigIntType.INSTANCE).args(IntegerType.INSTANCE),
+ FunctionSignature.ret(BigIntType.INSTANCE).args(BigIntType.INSTANCE),
+ FunctionSignature.ret(LargeIntType.INSTANCE).args(LargeIntType.INSTANCE)
+ );
+
+ private Expression buckets;
Review Comment:
bucket is not use anymore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] YangShaw commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "YangShaw (via GitHub)" <gi...@apache.org>.
YangShaw commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1098228130
##########
fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4:
##########
@@ -327,6 +332,32 @@ functionIdentifier
| LEFT | RIGHT
;
+windowSpec
+ : name=identifier
+ | LEFT_PAREN name=identifier RIGHT_PAREN
Review Comment:
haven't yet; i have commented this two line temporarily
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #14397: [feature](nereids)support window function
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #14397:
URL: https://github.com/apache/doris/pull/14397#discussion_r1092987998
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/WindowFunction.java:
##########
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.window;
+
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.BoundFunction;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.VarcharType;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * Window functions, as known as analytic functions.
+ */
+public abstract class WindowFunction extends BoundFunction {
+
+ public WindowFunction(String name, Expression... arguments) {
+ super(name, arguments);
+ }
+
+ public WindowFunction(String name, List<Expression> children) {
+ super(name, children);
+ }
+
+ protected List<DataType> intermediateTypes() {
+ return ImmutableList.of(VarcharType.SYSTEM_DEFAULT);
+ }
Review Comment:
window function do not need intermediateTypes i think
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/Ntile.java:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.window;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable;
+import org.apache.doris.nereids.trees.expressions.functions.CustomSignature;
+import org.apache.doris.nereids.trees.expressions.shape.LeafExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.IntegerType;
+
+/**
+ * window function: Ntile()
+ */
+public class Ntile extends WindowFunction implements LeafExpression, AlwaysNotNullable, CustomSignature {
Review Comment:
why use CustomSignature here?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalQuickSort.java:
##########
@@ -44,20 +44,19 @@ public PhysicalQuickSort(List<OrderKey> orderKeys,
}
/**
- * Constructor of PhysicalHashJoinNode.
+ * Constructor of PhysicalQuickSort.
Review Comment:
do not change the indent of this file~
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/FirstOrLastValue.java:
##########
@@ -0,0 +1,47 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.window;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.CustomSignature;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.types.DataType;
+
+/** parent class for first_value() and last_value() */
+public class FirstOrLastValue extends WindowFunction implements UnaryExpression, PropagateNullable, CustomSignature {
Review Comment:
abstract
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/Lag.java:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.window;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.ImplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.expressions.shape.TernaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.IntegerType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/** Window function: lag */
+public class Lag extends WindowFunction implements TernaryExpression, PropagateNullable, ImplicitlyCastableSignature {
+
+ public Lag(Expression child) {
+ this(child, Literal.of(1), Literal.of(null));
+ }
+
+ public Lag(Expression child, Expression offset) {
+ this(child, offset, Literal.of(null));
+ }
+
+ public Lag(Expression child, Expression offset, Expression defaultValue) {
+ super("lag", child, offset, defaultValue);
+ }
+
+ public Lag(List<Expression> children) {
+ super("lag", children);
+ }
+
+ public Expression getOffset() {
+ return child(1);
+ }
+
+ public Expression getDefaultValue() {
+ return child(2);
+ }
+
+ public void setDefaultValue(Expression defaultValue) {
+ this.children.set(2, defaultValue);
+ }
Review Comment:
our expression is immutable, so should use withChildren to update default value
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/Lead.java:
##########
@@ -0,0 +1,91 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.window;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.expressions.shape.TernaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.DataType;
+
+import com.google.common.base.Preconditions;
+
+import java.util.List;
+
+/**
+ * Window function: Lead()
+ */
+public class Lead extends WindowFunction implements TernaryExpression, PropagateNullable, ExplicitlyCastableSignature {
+
+ public Lead(Expression child) {
+ this(child, Literal.of(1), Literal.of(null));
+ }
+
+ public Lead(Expression child, Expression offset) {
+ this(child, offset, Literal.of(null));
+ }
+
+ public Lead(Expression child, Expression offset, Expression defaultValue) {
+ super("lead", child, offset, defaultValue);
+ }
+
+ public Lead(List<Expression> children) {
+ super("lead", children);
+ }
+
+ public Expression getOffset() {
+ return child(1);
+ }
+
+ public Expression getDefaultValue() {
+ return child(2);
+ }
+
+ public void setDefaultValue(Expression defaultValue) {
+ this.children.set(2, defaultValue);
+ }
+
+ @Override
+ public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
+ return visitor.visitLead(this, context);
+ }
+
+ @Override
+ public List<FunctionSignature> getSignatures() {
+ return null;
+ }
+
+ @Override
+ public FunctionSignature searchSignature(List<FunctionSignature> signatures) {
+ return null;
+ }
Review Comment:
why return null?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -611,4 +623,35 @@ private StatsDeriveResult computeGenerate(Generate generate) {
}
return new StatsDeriveResult(count, columnStatsMap);
}
+
+ private StatsDeriveResult computeWindow(Window windowOperator) {
+ StatsDeriveResult stats = groupExpression.childStatistics(0);
+ Map<Id, ColumnStatistic> columnStatisticMap = new HashMap<>();
+ Map<Id, ColumnStatistic> childColumnStats = stats.getSlotIdToColumnStats();
+ windowOperator.getOutputExpressions().stream()
+ .map(expr -> {
+ ColumnStatistic value = null;
+ Set<Slot> slots = expr.getInputSlots();
+ if (slots.isEmpty()) {
+ value = ColumnStatistic.DEFAULT;
+ } else {
+ for (Slot slot : slots) {
+ if (childColumnStats.containsKey(slot.getExprId())) {
+ value = childColumnStats.get(slot.getExprId());
+ break;
+ }
+ }
+ if (value == null) {
+ // process window expression
+ // todo: how to set stats?
+ // ColumnStatisticBuilder builder = new ColumnStatisticBuilder().setNdv(1)
+ // .setDataSize(expr.getDataType().width());
Review Comment:
remove useless comments, except TODO
##########
fe/fe-core/src/test/java/org/apache/doris/nereids/parser/NereidsParserTest.java:
##########
@@ -106,6 +106,16 @@ public void testParseCTE() {
Assertions.assertEquals(columnAliases.get().size(), 2);
}
+ @Test
+ public void testParseWindowFunctions() {
+ NereidsParser nereidsParser = new NereidsParser();
+ LogicalPlan logicalPlan;
+
+ String windowSql1 = "select rank() over(partition by k1 order by k1) as ranking from t1";
+ logicalPlan = nereidsParser.parseSingle(windowSql1);
+ System.out.println(logicalPlan.treeString());
Review Comment:
should we need to check plan pattern?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/Ntile.java:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.window;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable;
+import org.apache.doris.nereids.trees.expressions.functions.CustomSignature;
+import org.apache.doris.nereids.trees.expressions.shape.LeafExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.IntegerType;
+
+/**
+ * window function: Ntile()
+ */
+public class Ntile extends WindowFunction implements LeafExpression, AlwaysNotNullable, CustomSignature {
+
+ private int buckets;
+
+ public Ntile(int buckets) {
Review Comment:
why not treat buckets as its child?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -611,4 +623,35 @@ private StatsDeriveResult computeGenerate(Generate generate) {
}
return new StatsDeriveResult(count, columnStatsMap);
}
+
+ private StatsDeriveResult computeWindow(Window windowOperator) {
+ StatsDeriveResult stats = groupExpression.childStatistics(0);
+ Map<Id, ColumnStatistic> columnStatisticMap = new HashMap<>();
+ Map<Id, ColumnStatistic> childColumnStats = stats.getSlotIdToColumnStats();
+ windowOperator.getOutputExpressions().stream()
Review Comment:
should assign return value to `columnStatisticMap`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org