You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "jackwener (via GitHub)" <gi...@apache.org> on 2023/12/19 04:15:56 UTC
[PR] [feature](Nereids): eliminate groupby [doris]
jackwener opened a new pull request, #28615:
URL: https://github.com/apache/doris/pull/28615
## Proposed changes
eliminate groupby when unique constraint exist
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1870085883
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #28615:
URL: https://github.com/apache/doris/pull/28615#discussion_r1432514008
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FunctionalDependencies.java:
##########
@@ -220,7 +220,8 @@ public void replace(Map<Slot, Slot> replaceMap) {
.map(s -> replaceMap.getOrDefault(s, s))
.collect(Collectors.toSet());
slotSets = slotSets.stream()
- .map(set -> set.stream().map(replaceMap::get).collect(ImmutableSet.toImmutableSet()))
+ .map(set -> set.stream().map(s -> replaceMap.getOrDefault(s, s))
Review Comment:
what is it for ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #28615:
URL: https://github.com/apache/doris/pull/28615#discussion_r1432514962
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateGroupBy.java:
##########
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.IsNull;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Count;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Max;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Min;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.If;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.util.PlanUtils;
+
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Eliminate GroupBy.
+ */
+public class EliminateGroupBy extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalAggregate()
+ .when(agg -> agg.getGroupByExpressions().stream().allMatch(expr -> expr instanceof Slot))
+ .then(agg -> {
+ Set<Slot> groupby = agg.getGroupByExpressions().stream().map(e -> (Slot) e)
+ .collect(Collectors.toSet());
+ boolean unique = agg.child().getLogicalProperties().getFunctionalDependencies()
Review Comment:
pls also consider the uk model also, besides fd injection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1864030794
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1869460396
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit f610ba46376c16f4af29114f6e7d6b4363ef2f00, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4791 4486 4517 4486
q2 367 148 158 148
q3 1476 1328 1215 1215
q4 1174 1016 972 972
q5 3193 3197 3180 3180
q6 259 135 134 134
q7 1032 497 488 488
q8 2278 2296 2263 2263
q9 6693 6731 6712 6712
q10 3226 3322 3345 3322
q11 333 216 209 209
q12 360 206 209 206
q13 4556 3808 3783 3783
q14 244 215 216 215
q15 568 524 516 516
q16 437 383 387 383
q17 1044 796 645 645
q18 7118 6700 7603 6700
q19 1572 1609 1606 1606
q20 543 309 298 298
q21 3299 2783 2797 2783
q22 364 300 300 300
Total cold run time: 44927 ms
Total hot run time: 40564 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4432 4421 4423 4421
q2 278 170 169 169
q3 3527 3502 3502 3502
q4 2440 2413 2399 2399
q5 5753 5715 5735 5715
q6 253 126 131 126
q7 2384 1878 1891 1878
q8 3633 3631 3613 3613
q9 9058 8985 9006 8985
q10 3928 4052 4022 4022
q11 489 366 363 363
q12 774 626 610 610
q13 4278 3558 3554 3554
q14 298 261 252 252
q15 567 527 523 523
q16 486 458 446 446
q17 1974 1966 1960 1960
q18 8595 8240 8300 8240
q19 1824 1808 1827 1808
q20 2258 1944 1944 1944
q21 6622 6233 6209 6209
q22 566 476 475 475
Total cold run time: 64417 ms
Total hot run time: 61214 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1862406924
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on code in PR #28615:
URL: https://github.com/apache/doris/pull/28615#discussion_r1432823182
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateGroupBy.java:
##########
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.IsNull;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Count;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Max;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Min;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.If;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.util.PlanUtils;
+
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Eliminate GroupBy.
+ */
+public class EliminateGroupBy extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalAggregate()
+ .when(agg -> agg.getGroupByExpressions().stream().allMatch(expr -> expr instanceof Slot))
+ .then(agg -> {
+ Set<Slot> groupby = agg.getGroupByExpressions().stream().map(e -> (Slot) e)
+ .collect(Collectors.toSet());
+ boolean unique = agg.child().getLogicalProperties().getFunctionalDependencies()
Review Comment:
cc @XieJiann
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1863814765
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on code in PR #28615:
URL: https://github.com/apache/doris/pull/28615#discussion_r1436698246
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateGroupBy.java:
##########
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.IsNull;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Count;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Max;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Min;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.If;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.util.PlanUtils;
+
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Eliminate GroupBy.
+ */
+public class EliminateGroupBy extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalAggregate()
+ .when(agg -> agg.getGroupByExpressions().stream().allMatch(expr -> expr instanceof Slot))
+ .then(agg -> {
+ Set<Slot> groupby = agg.getGroupByExpressions().stream().map(e -> (Slot) e)
+ .collect(Collectors.toSet());
+ boolean unique = agg.child().getLogicalProperties().getFunctionalDependencies()
Review Comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1869930978
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #28615:
URL: https://github.com/apache/doris/pull/28615#discussion_r1432512615
##########
conf/fe.conf:
##########
@@ -69,6 +69,7 @@ arrow_flight_sql_port = -1
# If no ip match this rule, will choose one randomly.
# use CIDR format, e.g. 10.10.10.0/24 or IP format, e.g. 10.10.10.1
# Default value is empty.
+priority_networks = 127.0.0.1/24
Review Comment:
remove this.
##########
conf/be.conf:
##########
@@ -55,7 +55,7 @@ ssl_private_key_path = "$DORIS_HOME/conf/key.pem"
# If no ip match this rule, will choose one randomly.
# use CIDR format, e.g. 10.10.10.0/24 or IP format, e.g. 10.10.10.1
# Default value is empty.
-# priority_networks = 10.10.10.0/24;192.168.0.0/16
+priority_networks = 127.0.0.1/24
Review Comment:
remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on code in PR #28615:
URL: https://github.com/apache/doris/pull/28615#discussion_r1432816472
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/FunctionalDependencies.java:
##########
@@ -220,7 +220,8 @@ public void replace(Map<Slot, Slot> replaceMap) {
.map(s -> replaceMap.getOrDefault(s, s))
.collect(Collectors.toSet());
slotSets = slotSets.stream()
- .map(set -> set.stream().map(replaceMap::get).collect(ImmutableSet.toImmutableSet()))
+ .map(set -> set.stream().map(s -> replaceMap.getOrDefault(s, s))
Review Comment:
bugfix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1869876449
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1864013889
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1864614095
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1864681335
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.21 seconds
stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17184081310 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1869466152
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.39 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 20.7 seconds inserted 10000000 Rows, about 483K ops/s
storage size: 17183634681 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on code in PR #28615:
URL: https://github.com/apache/doris/pull/28615#discussion_r1430923364
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateGroupBy.java:
##########
@@ -0,0 +1,80 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.IsNull;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Count;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Max;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Min;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.If;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.util.PlanUtils;
+
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Eliminate GroupBy.
+ */
+public class EliminateGroupBy extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalAggregate()
+ .when(agg -> agg.getGroupByExpressions().stream().allMatch(expr -> expr instanceof Slot))
+ .then(agg -> {
+ Set<Slot> groupby = agg.getGroupByExpressions().stream().map(e -> (Slot) e)
+ .collect(Collectors.toSet());
+ boolean unique = agg.getLogicalProperties().getFunctionalDependencies().isUnique(groupby);
+ if (!unique) {
+ return null;
+ }
+ Set<AggregateFunction> aggregateFunctions = agg.getAggregateFunctions();
+ if (!aggregateFunctions.stream().allMatch(
+ f -> (f instanceof Sum || f instanceof Count || f instanceof Min || f instanceof Max)
+ && (f.arity() == 1 && f.child(0) instanceof Slot))) {
+ return null;
+ }
+
+ List<NamedExpression> newOutput = agg.getOutputExpressions().stream().map(ne -> {
+ if (ne instanceof Alias && ne.child(0) instanceof AggregateFunction) {
+ AggregateFunction f = (AggregateFunction) ne.child(0);
+ if (f instanceof Sum || f instanceof Min || f instanceof Max) {
+ return new Alias(ne.getExprId(), f.child(0), ne.getName());
+ } else if (f instanceof Count) {
+ return (NamedExpression) ne.withChildren(
+ new If(new IsNull(f.child(0)), Literal.of(0), Literal.of(1)));
Review Comment:
It is right.
such as for empty children.
```sql
create table test (id int, age int);
select min(id) from test group by age;
select id from test;
```
They are same;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1863876367
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.65 seconds
stream load tsv: 572 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17184116583 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1870851413
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1869485681
run External Regression
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on code in PR #28615:
URL: https://github.com/apache/doris/pull/28615#discussion_r1430898983
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateGroupBy.java:
##########
@@ -0,0 +1,80 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.IsNull;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Count;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Max;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Min;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.If;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.util.PlanUtils;
+
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Eliminate GroupBy.
+ */
+public class EliminateGroupBy extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalAggregate()
+ .when(agg -> agg.getGroupByExpressions().stream().allMatch(expr -> expr instanceof Slot))
+ .then(agg -> {
+ Set<Slot> groupby = agg.getGroupByExpressions().stream().map(e -> (Slot) e)
+ .collect(Collectors.toSet());
+ boolean unique = agg.getLogicalProperties().getFunctionalDependencies().isUnique(groupby);
+ if (!unique) {
Review Comment:
IsUniqueAndNotNull
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1870137051
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 36eb5af184f1b2276adf18e14323b804e354e7d0, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4967 4626 4629 4626
q2 385 169 157 157
q3 1457 1262 1198 1198
q4 1138 961 901 901
q5 3168 3163 3175 3163
q6 250 136 129 129
q7 1039 489 504 489
q8 2263 2252 2247 2247
q9 6731 6691 6693 6691
q10 3174 3277 3281 3277
q11 319 207 211 207
q12 347 203 204 203
q13 4160 3466 3455 3455
q14 240 221 215 215
q15 581 514 528 514
q16 435 390 381 381
q17 1040 791 560 560
q18 7118 6897 6880 6880
q19 1622 1632 1645 1632
q20 539 316 306 306
q21 3193 2723 2724 2723
q22 371 313 313 313
Total cold run time: 44537 ms
Total hot run time: 40267 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4594 4556 4568 4556
q2 271 167 164 164
q3 3370 3368 3358 3358
q4 2230 2213 2215 2213
q5 5715 5731 5719 5719
q6 241 116 117 116
q7 2381 1821 1858 1821
q8 3625 3610 3609 3609
q9 9022 9050 8965 8965
q10 3808 3900 3912 3900
q11 489 368 390 368
q12 776 592 596 592
q13 3933 3146 3182 3146
q14 289 264 253 253
q15 581 520 523 520
q16 512 441 437 437
q17 1984 1954 1962 1954
q18 8725 8204 8287 8204
q19 1742 1767 1773 1767
q20 2248 1960 1938 1938
q21 6149 5784 5793 5784
q22 565 455 492 455
Total cold run time: 63250 ms
Total hot run time: 59839 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1869900985
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit f7233ab480dadfc38dc090ca4995e08b42217f56, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4797 4473 4479 4473
q2 377 174 159 159
q3 1473 1267 1236 1236
q4 1151 1006 960 960
q5 3185 3179 3241 3179
q6 257 134 134 134
q7 1028 480 496 480
q8 2247 2264 2241 2241
q9 6751 6702 6673 6673
q10 3224 3281 3291 3281
q11 335 203 213 203
q12 359 218 210 210
q13 4526 3838 3779 3779
q14 245 210 218 210
q15 574 520 541 520
q16 438 385 381 381
q17 1040 798 607 607
q18 7096 6748 6761 6748
q19 1582 1601 1607 1601
q20 533 347 293 293
q21 3257 2778 2812 2778
q22 366 306 311 306
Total cold run time: 44841 ms
Total hot run time: 40452 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4423 4416 4401 4401
q2 271 167 172 167
q3 3516 3502 3494 3494
q4 2431 2421 2428 2421
q5 5728 5719 5694 5694
q6 247 125 126 125
q7 2367 1856 1851 1851
q8 3609 3598 3608 3598
q9 9040 8999 9012 8999
q10 3895 4022 4003 4003
q11 484 371 357 357
q12 764 624 606 606
q13 4301 3588 3580 3580
q14 288 245 268 245
q15 581 524 523 523
q16 491 467 454 454
q17 1972 1950 1930 1930
q18 8664 8077 8335 8077
q19 1819 1817 1822 1817
q20 2260 1954 1935 1935
q21 6617 6260 6253 6253
q22 537 487 485 485
Total cold run time: 64305 ms
Total hot run time: 61015 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1869896894
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.94 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.0 seconds inserted 10000000 Rows, about 357K ops/s
storage size: 17183300231 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1869384770
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1864077723
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener merged PR #28615:
URL: https://github.com/apache/doris/pull/28615
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1870088851
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1870124000
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17184053274 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1869443134
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on code in PR #28615:
URL: https://github.com/apache/doris/pull/28615#discussion_r1430899501
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateGroupBy.java:
##########
@@ -0,0 +1,80 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.IsNull;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Count;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Max;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Min;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.If;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.util.PlanUtils;
+
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Eliminate GroupBy.
+ */
+public class EliminateGroupBy extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalAggregate()
+ .when(agg -> agg.getGroupByExpressions().stream().allMatch(expr -> expr instanceof Slot))
+ .then(agg -> {
+ Set<Slot> groupby = agg.getGroupByExpressions().stream().map(e -> (Slot) e)
+ .collect(Collectors.toSet());
+ boolean unique = agg.getLogicalProperties().getFunctionalDependencies().isUnique(groupby);
+ if (!unique) {
+ return null;
+ }
+ Set<AggregateFunction> aggregateFunctions = agg.getAggregateFunctions();
+ if (!aggregateFunctions.stream().allMatch(
+ f -> (f instanceof Sum || f instanceof Count || f instanceof Min || f instanceof Max)
+ && (f.arity() == 1 && f.child(0) instanceof Slot))) {
+ return null;
+ }
+
+ List<NamedExpression> newOutput = agg.getOutputExpressions().stream().map(ne -> {
+ if (ne instanceof Alias && ne.child(0) instanceof AggregateFunction) {
+ AggregateFunction f = (AggregateFunction) ne.child(0);
+ if (f instanceof Sum || f instanceof Min || f instanceof Max) {
+ return new Alias(ne.getExprId(), f.child(0), ne.getName());
+ } else if (f instanceof Count) {
+ return (NamedExpression) ne.withChildren(
+ new If(new IsNull(f.child(0)), Literal.of(0), Literal.of(1)));
Review Comment:
It's wrong for empty children
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1862547554
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1863869373
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit caf1718e345347616151dc5fd83d1b7379c2d62e, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4736 4499 4473 4473
q2 375 178 159 159
q3 1466 1254 1248 1248
q4 1118 957 958 957
q5 3207 3161 3166 3161
q6 247 127 126 126
q7 967 483 487 483
q8 2228 2208 2209 2208
q9 6718 6693 6697 6693
q10 3238 3277 3268 3268
q11 316 193 190 190
q12 364 225 221 221
q13 4569 3819 3830 3819
q14 241 210 215 210
q15 562 518 523 518
q16 459 400 407 400
q17 1005 583 582 582
q18 7265 6891 6895 6891
q19 1514 1404 1436 1404
q20 534 338 318 318
q21 3121 2698 2673 2673
q22 361 292 298 292
Total cold run time: 44611 ms
Total hot run time: 40294 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4379 4422 4410 4410
q2 271 172 177 172
q3 3552 3536 3541 3536
q4 2397 2394 2392 2392
q5 5760 5773 5773 5773
q6 241 121 119 119
q7 2405 1918 1902 1902
q8 3539 3550 3559 3550
q9 9041 9046 9017 9017
q10 3937 4006 4046 4006
q11 487 379 386 379
q12 778 609 613 609
q13 4329 3590 3571 3571
q14 295 248 266 248
q15 572 517 516 516
q16 529 452 486 452
q17 1894 1871 1872 1871
q18 8621 8183 8310 8183
q19 1724 1771 1785 1771
q20 2267 1964 1944 1944
q21 6606 6222 6223 6222
q22 530 458 427 427
Total cold run time: 64154 ms
Total hot run time: 61070 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1863866567
run feut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #28615:
URL: https://github.com/apache/doris/pull/28615#discussion_r1432514962
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateGroupBy.java:
##########
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.IsNull;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Count;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Max;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Min;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.If;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.util.PlanUtils;
+
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Eliminate GroupBy.
+ */
+public class EliminateGroupBy extends OneRewriteRuleFactory {
+
+ @Override
+ public Rule build() {
+ return logicalAggregate()
+ .when(agg -> agg.getGroupByExpressions().stream().allMatch(expr -> expr instanceof Slot))
+ .then(agg -> {
+ Set<Slot> groupby = agg.getGroupByExpressions().stream().map(e -> (Slot) e)
+ .collect(Collectors.toSet());
+ boolean unique = agg.child().getLogicalProperties().getFunctionalDependencies()
Review Comment:
will it work if the uk info is injected by constraint?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Nereids): eliminate groupby [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28615:
URL: https://github.com/apache/doris/pull/28615#issuecomment-1864654876
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 69588a29ee89ca95f824a2d7005acf42bfd88b2f, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4684 4435 4469 4435
q2 364 149 158 149
q3 1476 1267 1199 1199
q4 1122 942 916 916
q5 3163 3171 3169 3169
q6 246 129 128 128
q7 991 496 477 477
q8 2190 2193 2177 2177
q9 6672 6660 6674 6660
q10 3250 3257 3295 3257
q11 305 185 188 185
q12 354 208 212 208
q13 4582 3793 3802 3793
q14 241 222 219 219
q15 576 522 531 522
q16 452 399 393 393
q17 1008 561 567 561
q18 7185 6916 6968 6916
q19 1516 1355 1435 1355
q20 503 322 290 290
q21 3073 2581 2687 2581
q22 361 294 298 294
Total cold run time: 44314 ms
Total hot run time: 39884 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4397 4371 4381 4371
q2 275 170 181 170
q3 3543 3542 3549 3542
q4 2391 2381 2384 2381
q5 5773 5785 5764 5764
q6 239 120 125 120
q7 2379 1872 1858 1858
q8 3516 3520 3536 3520
q9 9065 9042 9014 9014
q10 3951 4004 4012 4004
q11 495 394 385 385
q12 786 600 602 600
q13 4313 3582 3632 3582
q14 292 258 272 258
q15 574 518 519 518
q16 514 483 474 474
q17 1886 1873 1884 1873
q18 8715 8329 8338 8329
q19 1715 1723 1739 1723
q20 2268 1969 1958 1958
q21 6541 6247 6223 6223
q22 526 441 460 441
Total cold run time: 64154 ms
Total hot run time: 61108 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org