You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2023/01/06 03:19:00 UTC
[GitHub] [doris] 924060929 opened a new pull request, #15671: [Enhancement](Nereids) Generate aggregate functions
924060929 opened a new pull request, #15671:
URL: https://github.com/apache/doris/pull/15671
# Proposed changes
1. generate lots of aggregate functions
2. support array functions, like `array_union`. we should support array grammar in the future, e.g. `select [1, 2, 3]`
3. add `checkLegality` function to check the legality of expression before do type coercion, remove the `ForbiddenMetricTypeArguments`
4. refactor the `NullableAggregateFunction`: distinct is the first parameter, alwaysNullable is the second parameter; Fix some wrong initialize order: super(distinct, alwaysNullable) and super(alwaysNullable, distinct)
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [x] No
- [ ] I don't know
5. Has unit tests been added:
- [x] Yes
- [ ] No
- [ ] No Need
6. Has document been added or modified:
- [ ] Yes
- [ ] No
- [x] No Need
7. Does it need to update dependencies:
- [ ] Yes
- [x] No
8. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [x] No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #15671: [feature](Nereids) Support lots of aggregate functions
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #15671:
URL: https://github.com/apache/doris/pull/15671#discussion_r1064463906
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java:
##########
@@ -57,6 +59,27 @@ public abstract class DataType implements AbstractDataType {
.put(FloatType.class, () -> DoubleType.INSTANCE)
.build();
+ private static final Map<Class<? extends DataType>, Promotion<DataType>> FULL_PRIMITIVE_TYPE_PROMOTION_MAP
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #15671: [feature](Nereids) Support lots of aggregate functions
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #15671:
URL: https://github.com/apache/doris/pull/15671#discussion_r1064320710
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java:
##########
@@ -57,6 +59,27 @@ public abstract class DataType implements AbstractDataType {
.put(FloatType.class, () -> DoubleType.INSTANCE)
.build();
+ private static final Map<Class<? extends DataType>, Promotion<DataType>> FULL_PRIMITIVE_TYPE_PROMOTION_MAP
Review Comment:
need some comment to explain the mean of ordinal in promotion list
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -1551,4 +1559,21 @@ public DataType visitPrimitiveDataType(PrimitiveDataTypeContext ctx) {
ctx.INTEGER_VALUE().stream().map(ParseTree::getText).forEach(l::add);
return DataType.convertPrimitiveFromStrings(l);
}
+
+ private Expression parseFunctionWithOrderKeys(String functionName, boolean isDistinct,
Review Comment:
could we add a new interface named `supportOrderKeys` and offer some helper methold such as `withOrderKeys`, `getOrderkeys`?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/AvgWeighted.java:
##########
@@ -0,0 +1,87 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.agg;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.shape.BinaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.DecimalV2Type;
+import org.apache.doris.nereids.types.DoubleType;
+import org.apache.doris.nereids.types.FloatType;
+import org.apache.doris.nereids.types.IntegerType;
+import org.apache.doris.nereids.types.SmallIntType;
+import org.apache.doris.nereids.types.TinyIntType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * AggregateFunction 'avg_weighted'. This class is generated by GenerateFunction.
+ */
+public class AvgWeighted extends AggregateFunction
+ implements BinaryExpression, ExplicitlyCastableSignature, PropagateNullable {
Review Comment:
all PropagateNullable aggregate function should extends from NullableAggregateFunction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #15671: [feature](Nereids) Support lots of aggregate functions
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #15671:
URL: https://github.com/apache/doris/pull/15671#discussion_r1064456681
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/AvgWeighted.java:
##########
@@ -0,0 +1,87 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.agg;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.shape.BinaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.DecimalV2Type;
+import org.apache.doris.nereids.types.DoubleType;
+import org.apache.doris.nereids.types.FloatType;
+import org.apache.doris.nereids.types.IntegerType;
+import org.apache.doris.nereids.types.SmallIntType;
+import org.apache.doris.nereids.types.TinyIntType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * AggregateFunction 'avg_weighted'. This class is generated by GenerateFunction.
+ */
+public class AvgWeighted extends AggregateFunction
+ implements BinaryExpression, ExplicitlyCastableSignature, PropagateNullable {
Review Comment:
I think we should refactor NullableAggregateFunction in the next PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #15671: [feature](Nereids) Support lots of aggregate functions
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #15671:
URL: https://github.com/apache/doris/pull/15671#discussion_r1064456325
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -1551,4 +1559,21 @@ public DataType visitPrimitiveDataType(PrimitiveDataTypeContext ctx) {
ctx.INTEGER_VALUE().stream().map(ParseTree::getText).forEach(l::add);
return DataType.convertPrimitiveFromStrings(l);
}
+
+ private Expression parseFunctionWithOrderKeys(String functionName, boolean isDistinct,
Review Comment:
currently we only support the group_concat, we can consider generality if more function need order by
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #15671: [feature](Nereids) Support lots of aggregate functions
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #15671:
URL: https://github.com/apache/doris/pull/15671#discussion_r1064456325
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -1551,4 +1559,21 @@ public DataType visitPrimitiveDataType(PrimitiveDataTypeContext ctx) {
ctx.INTEGER_VALUE().stream().map(ParseTree::getText).forEach(l::add);
return DataType.convertPrimitiveFromStrings(l);
}
+
+ private Expression parseFunctionWithOrderKeys(String functionName, boolean isDistinct,
Review Comment:
currently we only support the group_concat, we can consider generality if more functions need order by
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15671: [feature](Nereids) Support lots of aggregate functions
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #15671:
URL: https://github.com/apache/doris/pull/15671#issuecomment-1373118890
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 37.96 seconds
load time: 516 seconds
storage size: 17122043228 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230106040929_clickbench_pr_74565.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #15671: [feature](Nereids) Support lots of aggregate functions
Posted by GitBox <gi...@apache.org>.
morrySnow merged PR #15671:
URL: https://github.com/apache/doris/pull/15671
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org