You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "XuPengfei-1020 (via GitHub)" <gi...@apache.org> on 2024/06/14 12:18:24 UTC
[PR] New expression extractor for partitions pruning. [doris]
XuPengfei-1020 opened a new pull request, #36326:
URL: https://github.com/apache/doris/pull/36326
## Proposed changes
Original extractor(TryEliminateUninterestedPredicates.java) caused some errors while try to derive the expressions which can be used for pruning partitions.
I tried to write a new extractor(and with unit tests) for pruning partitions, it is more simple and reliable (I think).
The theory of extractor is pretty simple:
A:Sort the expression in two kinds:
1. evaluable-expression (let's mark it as E).
Expressions that can be evaluated in the partition pruning stage.
In the other word: not contains non-partition slots or deterministic expression.
2. un-evaluable-expression (let's mark it as UE).
Expressions that can NOT be evaluated in the partition pruning stage.
In the other word: contains non-partition slots or deterministic expression.
B: Travel the predicate, only point on AND and OR operator, following the rule:
(E and UE) -> (E and TRUE) -> E
(UE and UE) -> TRUE
(E and E) -> (E and E)
(E or UE) -> TRUE
(UE or UE) -> TRUE
(E or E) -> (E or E)
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] New expression extractor for partitions pruning. [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2172083476
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]
Posted by "XuPengfei-1020 (via GitHub)" <gi...@apache.org>.
XuPengfei-1020 commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2172987456
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] New expression extractor for partitions pruning. [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #36326:
URL: https://github.com/apache/doris/pull/36326#discussion_r1642139159
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruneExpressionExtractor.java:
##########
@@ -0,0 +1,204 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.rules.expression.rules.PartitionPruneExpressionExtractor.Context;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SubqueryExpr;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * PartitionPruneExpressionExtractor
+ *
+ * This rewriter only used to extract the expression that can be used in partition pruning from
+ * the whole predicate expression.
+ * The theory of extractor is pretty simple:
+ * A:Sort the expression in two kinds:
+ * 1. evaluable-expression (let's mark it as E).
+ * Expressions that can be evaluated in the partition pruning stage.
+ * In the other word: not contains non-partition slots or deterministic expression.
+ * 2. un-evaluable-expression (let's mark it as UE).
+ * Expressions that can NOT be evaluated in the partition pruning stage.
+ * In the other word: contains non-partition slots or deterministic expression.
+ *
+ * B: Travel the predicate, only point on AND and OR operator, following the rule:
+ * (E and UE) -> (E and TRUE) -> E
+ * (UE and UE) -> TRUE
+ * (E and E) -> (E and E)
+ * (E or UE) -> TRUE
+ * (UE or UE) -> TRUE
+ * (E or E) -> (E or E)
+ *
+ * e.g.
+ * (part = 1 and non_part = 'a') or (part = 2)
+ * -> (part = 1 and true) or (part = 2)
+ * -> (part = 1) or (part = 2)
+ *
+ * It's better that do some expression optimize(like fold, eliminate etc.) on predicate before this step.
+ */
+public class PartitionPruneExpressionExtractor extends DefaultExpressionRewriter<Context> {
+ private final ExpressionEvaluableDetector expressionEvaluableDetector;
+
+ private PartitionPruneExpressionExtractor(Set<Slot> interestedSlots, CascadesContext cascadesContext) {
+ this.expressionEvaluableDetector = new ExpressionEvaluableDetector(interestedSlots);
+ }
+
+ /**
+ * Extract partition prune expression from predicate
+ */
+ public static Expression extract(Expression predicate,
+ Set<Slot> partitionSlots,
+ CascadesContext cascadesContext) {
+ predicate = predicate.accept(FoldConstantRuleOnFE.VISITOR_INSTANCE, null);
Review Comment:
new an ExpressionRewriteContext as context parameter
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruneExpressionExtractor.java:
##########
@@ -0,0 +1,204 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.rules.expression.rules.PartitionPruneExpressionExtractor.Context;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SubqueryExpr;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * PartitionPruneExpressionExtractor
+ *
+ * This rewriter only used to extract the expression that can be used in partition pruning from
+ * the whole predicate expression.
+ * The theory of extractor is pretty simple:
+ * A:Sort the expression in two kinds:
+ * 1. evaluable-expression (let's mark it as E).
+ * Expressions that can be evaluated in the partition pruning stage.
+ * In the other word: not contains non-partition slots or deterministic expression.
+ * 2. un-evaluable-expression (let's mark it as UE).
+ * Expressions that can NOT be evaluated in the partition pruning stage.
+ * In the other word: contains non-partition slots or deterministic expression.
+ *
+ * B: Travel the predicate, only point on AND and OR operator, following the rule:
+ * (E and UE) -> (E and TRUE) -> E
+ * (UE and UE) -> TRUE
+ * (E and E) -> (E and E)
+ * (E or UE) -> TRUE
+ * (UE or UE) -> TRUE
+ * (E or E) -> (E or E)
+ *
+ * e.g.
+ * (part = 1 and non_part = 'a') or (part = 2)
+ * -> (part = 1 and true) or (part = 2)
+ * -> (part = 1) or (part = 2)
+ *
+ * It's better that do some expression optimize(like fold, eliminate etc.) on predicate before this step.
+ */
+public class PartitionPruneExpressionExtractor extends DefaultExpressionRewriter<Context> {
+ private final ExpressionEvaluableDetector expressionEvaluableDetector;
+
+ private PartitionPruneExpressionExtractor(Set<Slot> interestedSlots, CascadesContext cascadesContext) {
+ this.expressionEvaluableDetector = new ExpressionEvaluableDetector(interestedSlots);
+ }
+
+ /**
+ * Extract partition prune expression from predicate
+ */
+ public static Expression extract(Expression predicate,
+ Set<Slot> partitionSlots,
+ CascadesContext cascadesContext) {
+ predicate = predicate.accept(FoldConstantRuleOnFE.VISITOR_INSTANCE, null);
+ PartitionPruneExpressionExtractor rewriter =
+ new PartitionPruneExpressionExtractor(partitionSlots, cascadesContext);
+ Context context = new Context();
+ Expression partitionPruneExpression = predicate.accept(rewriter, context);
+ if (context.containsUnEvaluableExpression) {
+ return BooleanLiteral.TRUE;
+ }
+ return partitionPruneExpression;
+ }
+
+ @Override
+ public Expression visit(Expression originExpr, Context parentContext) {
+ if (originExpr instanceof And) {
+ return this.visitAnd((And) originExpr, parentContext);
+ }
+ if (originExpr instanceof Or) {
+ return this.visitOr((Or) originExpr, parentContext);
+ }
+
+ return originExpr.accept(expressionEvaluableDetector, parentContext);
+ }
+
+ @Override
+ public Expression visitAnd(And node, Context parentContext) {
+ // handle left node
+ Context leftContext = new Context();
+ Expression newLeft = node.left().accept(this, leftContext);
+ // handle right node
+ Context rightContext = new Context();
+ Expression newRight = node.right().accept(this, rightContext);
+
+ // if anyone of them is FALSE, the whole expression should be FALSE.
+ if (newLeft == BooleanLiteral.FALSE || newRight == BooleanLiteral.FALSE) {
+ return BooleanLiteral.FALSE;
+ }
+
+ // If left node contains non-partition slot or is TURE, just discard it.
+ if (newLeft == BooleanLiteral.TRUE || leftContext.containsUnEvaluableExpression) {
+ return rightContext.containsUnEvaluableExpression ? BooleanLiteral.TRUE : newRight;
+ }
+
+ // If right node contains non-partition slot or is TURE, just discard it.
+ if (newRight == BooleanLiteral.TRUE || rightContext.containsUnEvaluableExpression) {
+ return newLeft;
+ }
+
+ // both does not contains non-partition slot.
+ return new And(newLeft, newRight);
+ }
+
+ @Override
+ public Expression visitOr(Or node, Context parentContext) {
+ // handle left node
+ Context leftContext = new Context();
+ Expression newLeft = node.left().accept(this, leftContext);
+ // handle right node
+ Context rightContext = new Context();
+ Expression newRight = node.right().accept(this, rightContext);
+
+ // if anyone of them is TRUE or contains non-partition slot, just return TRUE.
+ if (newLeft == BooleanLiteral.TRUE || newRight == BooleanLiteral.TRUE
+ || leftContext.containsUnEvaluableExpression || rightContext.containsUnEvaluableExpression) {
+ return BooleanLiteral.TRUE;
+ }
+
+ return new Or(newLeft, newRight);
+ }
+
+ /**
+ * Context
+ */
+ @VisibleForTesting
+ public static class Context {
+ private boolean containsUnEvaluableExpression;
+ }
+
+ /**
+ * The detector only indicate that whether a predicate contains interested slots or not,
+ * and do not change the predicate.
+ */
+ @VisibleForTesting
+ public static class ExpressionEvaluableDetector extends DefaultExpressionRewriter<Context> {
+ private final Set<Slot> partitionSlots;
+
+ public ExpressionEvaluableDetector(Set<Slot> partitionSlots) {
+ this.partitionSlots = Objects.requireNonNull(partitionSlots);
Review Comment:
and a simple exception message here, e.g.
```java
this.partitionSlots = Objects.requireNonNull(partitionSlots, "partitionSlots can not be null);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173104650
<details>
<summary>TPC-DS: <b>Total hot run time: 174163 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b0a828dcb1cfd5b422c86356f5ee5b1c4d534a54, data reload: false
query1 924 383 383 383
query2 6455 2368 2323 2323
query3 6638 205 209 205
query4 22524 17413 17476 17413
query5 3584 478 470 470
query6 244 160 167 160
query7 4607 304 286 286
query8 319 298 282 282
query9 8451 2392 2393 2392
query10 556 299 288 288
query11 10536 10147 9935 9935
query12 116 90 84 84
query13 1629 362 362 362
query14 10031 7596 7740 7596
query15 255 200 207 200
query16 7779 272 284 272
query17 1564 542 543 542
query18 2020 295 286 286
query19 208 175 167 167
query20 93 85 83 83
query21 206 129 143 129
query22 4647 4424 4374 4374
query23 34031 33649 33551 33551
query24 10161 2954 2920 2920
query25 572 353 353 353
query26 701 149 165 149
query27 2191 310 320 310
query28 5824 2071 2046 2046
query29 854 618 610 610
query30 234 152 149 149
query31 950 719 737 719
query32 90 50 55 50
query33 640 279 269 269
query34 845 472 466 466
query35 706 607 621 607
query36 1057 964 958 958
query37 140 67 70 67
query38 2870 2727 2746 2727
query39 852 802 860 802
query40 206 126 121 121
query41 54 49 49 49
query42 115 99 95 95
query43 600 542 547 542
query44 1083 727 722 722
query45 191 160 162 160
query46 1074 719 711 711
query47 1885 1767 1756 1756
query48 363 293 305 293
query49 841 395 417 395
query50 760 396 395 395
query51 6787 6729 6703 6703
query52 98 96 91 91
query53 361 287 296 287
query54 851 446 430 430
query55 73 75 75 75
query56 291 258 263 258
query57 1131 1051 1067 1051
query58 251 248 238 238
query59 3372 3373 3156 3156
query60 289 256 284 256
query61 96 90 92 90
query62 590 451 454 451
query63 313 286 289 286
query64 8452 2296 1752 1752
query65 3159 3081 3105 3081
query66 734 328 327 327
query67 15142 15366 14873 14873
query68 4548 558 538 538
query69 458 316 305 305
query70 1173 1028 1137 1028
query71 371 279 271 271
query72 6977 5519 5477 5477
query73 740 327 323 323
query74 5928 5515 5392 5392
query75 3363 2638 2706 2638
query76 2184 943 913 913
query77 409 304 304 304
query78 10651 9857 9716 9716
query79 2329 521 519 519
query80 1129 529 455 455
query81 567 223 222 222
query82 740 102 107 102
query83 224 169 178 169
query84 237 88 90 88
query85 1377 290 271 271
query86 487 305 291 291
query87 3291 3099 3078 3078
query88 4344 2362 2358 2358
query89 465 374 387 374
query90 1704 200 195 195
query91 128 101 101 101
query92 58 50 50 50
query93 1941 509 502 502
query94 1119 193 186 186
query95 398 316 314 314
query96 580 269 264 264
query97 3261 3052 3058 3052
query98 218 199 188 188
query99 1204 871 850 850
Total cold run time: 267869 ms
Total hot run time: 174163 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] New expression extractor for partitions pruning. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2172152606
<details>
<summary>TPC-H: <b>Total hot run time: 39833 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 91e523aa1074b21dfe00fcc978f6af9167fc70fb, data reload: false
------ Round 1 ----------------------------------
q1 17616 4401 4314 4314
q2 2015 194 193 193
q3 10457 1091 1172 1091
q4 10198 834 793 793
q5 7482 2616 2659 2616
q6 226 137 137 137
q7 953 599 590 590
q8 9227 2085 2090 2085
q9 8978 6516 6495 6495
q10 9011 3688 3682 3682
q11 479 241 240 240
q12 452 223 224 223
q13 18785 2971 2990 2971
q14 260 218 223 218
q15 518 476 473 473
q16 523 368 378 368
q17 975 664 692 664
q18 8014 7558 7350 7350
q19 7458 1509 1465 1465
q20 665 323 336 323
q21 4885 3212 4049 3212
q22 401 330 345 330
Total cold run time: 119578 ms
Total hot run time: 39833 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4445 4255 4232 4232
q2 362 273 263 263
q3 3075 2916 2859 2859
q4 1985 1794 1759 1759
q5 5524 5476 5422 5422
q6 230 140 141 140
q7 2228 1841 1823 1823
q8 3288 3427 3432 3427
q9 8655 8789 8721 8721
q10 4192 3712 3766 3712
q11 609 478 496 478
q12 783 659 646 646
q13 16995 3185 3153 3153
q14 310 272 264 264
q15 541 489 481 481
q16 491 428 426 426
q17 1812 1492 1477 1477
q18 8163 7957 7764 7764
q19 1859 1664 1460 1460
q20 2197 1873 1858 1858
q21 9146 4912 4740 4740
q22 662 559 552 552
Total cold run time: 77552 ms
Total hot run time: 55657 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173394738
<details>
<summary>TPC-H: <b>Total hot run time: 39849 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0c072d3adaecd053725e993dce322a63de6a8e9d, data reload: false
------ Round 1 ----------------------------------
q1 17632 4358 4257 4257
q2 2043 184 190 184
q3 10475 1141 1146 1141
q4 10190 855 819 819
q5 7467 2662 2653 2653
q6 226 136 135 135
q7 963 616 601 601
q8 9287 2044 2089 2044
q9 9451 6489 6467 6467
q10 9052 3779 3745 3745
q11 437 234 239 234
q12 437 238 231 231
q13 17800 2984 2991 2984
q14 266 212 212 212
q15 512 473 471 471
q16 513 392 377 377
q17 963 727 655 655
q18 8033 7405 7469 7405
q19 5334 1400 1447 1400
q20 644 321 325 321
q21 4890 3181 3989 3181
q22 394 337 332 332
Total cold run time: 117009 ms
Total hot run time: 39849 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4363 4226 4217 4217
q2 375 263 251 251
q3 2987 2917 2896 2896
q4 2004 1756 1749 1749
q5 5604 5521 5494 5494
q6 224 138 140 138
q7 2198 1948 1870 1870
q8 3303 3456 3401 3401
q9 8774 8744 8767 8744
q10 4169 3834 3835 3834
q11 604 507 488 488
q12 808 675 657 657
q13 15945 3176 3188 3176
q14 302 261 286 261
q15 530 477 489 477
q16 488 433 433 433
q17 1821 1504 1489 1489
q18 8179 7881 7880 7880
q19 1787 1562 1651 1562
q20 2151 1909 1880 1880
q21 8089 5029 4760 4760
q22 641 586 547 547
Total cold run time: 75346 ms
Total hot run time: 56204 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173111095
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] New expression extractor for partitions pruning. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2172182669
<details>
<summary>ClickBench: <b>Total hot run time: 30.49 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 91e523aa1074b21dfe00fcc978f6af9167fc70fb, data reload: false
query1 0.04 0.03 0.03
query2 0.08 0.04 0.04
query3 0.23 0.05 0.05
query4 1.67 0.06 0.06
query5 0.49 0.48 0.47
query6 1.14 0.72 0.72
query7 0.02 0.01 0.02
query8 0.05 0.04 0.05
query9 0.54 0.49 0.48
query10 0.54 0.52 0.54
query11 0.15 0.12 0.12
query12 0.14 0.12 0.12
query13 0.60 0.58 0.60
query14 0.76 0.80 0.78
query15 0.83 0.81 0.81
query16 0.36 0.35 0.36
query17 0.94 0.98 1.00
query18 0.22 0.24 0.25
query19 1.82 1.67 1.84
query20 0.01 0.02 0.02
query21 15.41 0.65 0.64
query22 4.32 6.66 2.14
query23 18.29 1.33 1.18
query24 2.24 0.21 0.21
query25 0.15 0.07 0.10
query26 0.29 0.18 0.17
query27 0.07 0.08 0.08
query28 13.20 1.01 0.99
query29 12.58 3.31 3.25
query30 0.25 0.07 0.06
query31 2.85 0.40 0.39
query32 3.26 0.49 0.47
query33 2.87 2.88 2.92
query34 16.92 4.44 4.39
query35 4.46 4.43 4.47
query36 0.65 0.47 0.46
query37 0.18 0.16 0.15
query38 0.15 0.14 0.15
query39 0.04 0.04 0.04
query40 0.16 0.14 0.14
query41 0.10 0.05 0.04
query42 0.06 0.05 0.04
query43 0.05 0.04 0.04
Total cold run time: 109.18 s
Total hot run time: 30.49 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173419624
<details>
<summary>TPC-DS: <b>Total hot run time: 171456 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0c072d3adaecd053725e993dce322a63de6a8e9d, data reload: false
query1 923 388 382 382
query2 7415 2352 2247 2247
query3 6647 214 208 208
query4 19032 17378 17141 17141
query5 3549 473 476 473
query6 237 180 154 154
query7 4576 293 294 293
query8 314 262 266 262
query9 8507 2391 2392 2391
query10 580 296 277 277
query11 10468 10011 10011 10011
query12 135 82 87 82
query13 1630 373 372 372
query14 10016 7607 7663 7607
query15 231 185 191 185
query16 7729 265 265 265
query17 1897 559 509 509
query18 1939 270 272 270
query19 202 160 160 160
query20 94 92 83 83
query21 215 134 132 132
query22 4345 4042 3995 3995
query23 33899 33653 34046 33653
query24 10715 2993 2976 2976
query25 582 395 385 385
query26 715 158 162 158
query27 2308 333 324 324
query28 5894 2069 2092 2069
query29 880 623 634 623
query30 235 154 156 154
query31 971 778 759 759
query32 93 52 58 52
query33 670 290 285 285
query34 885 489 473 473
query35 772 648 630 630
query36 1124 957 992 957
query37 146 73 73 73
query38 2997 2817 2839 2817
query39 912 833 867 833
query40 217 130 131 130
query41 55 51 53 51
query42 117 105 104 104
query43 590 549 521 521
query44 1082 716 726 716
query45 196 161 161 161
query46 1080 723 716 716
query47 1834 1759 1762 1759
query48 389 292 294 292
query49 827 432 410 410
query50 765 390 391 390
query51 6870 6730 6609 6609
query52 98 87 85 85
query53 360 293 292 292
query54 870 449 448 448
query55 73 73 74 73
query56 270 268 253 253
query57 1119 1035 1051 1035
query58 249 237 250 237
query59 3371 3026 3011 3011
query60 294 271 279 271
query61 106 90 96 90
query62 635 457 440 440
query63 323 286 295 286
query64 8612 2400 1766 1766
query65 3201 3102 3098 3098
query66 745 330 327 327
query67 15433 15073 15046 15046
query68 4511 537 549 537
query69 569 461 379 379
query70 1106 1157 1070 1070
query71 441 271 279 271
query72 7306 5786 2740 2740
query73 752 330 332 330
query74 5901 5497 5550 5497
query75 3382 2664 2689 2664
query76 2683 982 927 927
query77 596 310 298 298
query78 10375 9917 9809 9809
query79 2780 514 509 509
query80 2065 469 457 457
query81 597 216 226 216
query82 1001 116 107 107
query83 304 169 163 163
query84 262 91 83 83
query85 1047 279 279 279
query86 452 308 336 308
query87 3298 3118 3100 3100
query88 3704 2349 2335 2335
query89 469 374 391 374
query90 1596 194 193 193
query91 129 103 99 99
query92 60 47 50 47
query93 1951 493 496 493
query94 1067 189 184 184
query95 403 304 312 304
query96 589 267 266 266
query97 3278 3095 3046 3046
query98 226 206 195 195
query99 1350 856 839 839
Total cold run time: 268510 ms
Total hot run time: 171456 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #36326:
URL: https://github.com/apache/doris/pull/36326
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] New expression extractor for partitions pruning. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2167901580
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] New expression extractor for partitions pruning. [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #36326:
URL: https://github.com/apache/doris/pull/36326#discussion_r1642134171
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruneExpressionExtractor.java:
##########
@@ -0,0 +1,204 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.rules.expression.rules.PartitionPruneExpressionExtractor.Context;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SubqueryExpr;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * PartitionPruneExpressionExtractor
+ *
+ * This rewriter only used to extract the expression that can be used in partition pruning from
+ * the whole predicate expression.
+ * The theory of extractor is pretty simple:
+ * A:Sort the expression in two kinds:
+ * 1. evaluable-expression (let's mark it as E).
+ * Expressions that can be evaluated in the partition pruning stage.
+ * In the other word: not contains non-partition slots or deterministic expression.
+ * 2. un-evaluable-expression (let's mark it as UE).
+ * Expressions that can NOT be evaluated in the partition pruning stage.
+ * In the other word: contains non-partition slots or deterministic expression.
+ *
+ * B: Travel the predicate, only point on AND and OR operator, following the rule:
+ * (E and UE) -> (E and TRUE) -> E
+ * (UE and UE) -> TRUE
+ * (E and E) -> (E and E)
+ * (E or UE) -> TRUE
+ * (UE or UE) -> TRUE
+ * (E or E) -> (E or E)
+ *
+ * e.g.
+ * (part = 1 and non_part = 'a') or (part = 2)
+ * -> (part = 1 and true) or (part = 2)
+ * -> (part = 1) or (part = 2)
+ *
+ * It's better that do some expression optimize(like fold, eliminate etc.) on predicate before this step.
+ */
+public class PartitionPruneExpressionExtractor extends DefaultExpressionRewriter<Context> {
+ private final ExpressionEvaluableDetector expressionEvaluableDetector;
+
+ private PartitionPruneExpressionExtractor(Set<Slot> interestedSlots, CascadesContext cascadesContext) {
+ this.expressionEvaluableDetector = new ExpressionEvaluableDetector(interestedSlots);
+ }
+
+ /**
+ * Extract partition prune expression from predicate
+ */
+ public static Expression extract(Expression predicate,
+ Set<Slot> partitionSlots,
+ CascadesContext cascadesContext) {
+ predicate = predicate.accept(FoldConstantRuleOnFE.VISITOR_INSTANCE, null);
+ PartitionPruneExpressionExtractor rewriter =
+ new PartitionPruneExpressionExtractor(partitionSlots, cascadesContext);
+ Context context = new Context();
+ Expression partitionPruneExpression = predicate.accept(rewriter, context);
+ if (context.containsUnEvaluableExpression) {
+ return BooleanLiteral.TRUE;
+ }
+ return partitionPruneExpression;
+ }
+
+ @Override
+ public Expression visit(Expression originExpr, Context parentContext) {
+ if (originExpr instanceof And) {
+ return this.visitAnd((And) originExpr, parentContext);
+ }
+ if (originExpr instanceof Or) {
+ return this.visitOr((Or) originExpr, parentContext);
+ }
+
+ return originExpr.accept(expressionEvaluableDetector, parentContext);
+ }
+
+ @Override
+ public Expression visitAnd(And node, Context parentContext) {
+ // handle left node
+ Context leftContext = new Context();
+ Expression newLeft = node.left().accept(this, leftContext);
+ // handle right node
+ Context rightContext = new Context();
+ Expression newRight = node.right().accept(this, rightContext);
+
+ // if anyone of them is FALSE, the whole expression should be FALSE.
+ if (newLeft == BooleanLiteral.FALSE || newRight == BooleanLiteral.FALSE) {
+ return BooleanLiteral.FALSE;
+ }
+
+ // If left node contains non-partition slot or is TURE, just discard it.
+ if (newLeft == BooleanLiteral.TRUE || leftContext.containsUnEvaluableExpression) {
+ return rightContext.containsUnEvaluableExpression ? BooleanLiteral.TRUE : newRight;
+ }
+
+ // If right node contains non-partition slot or is TURE, just discard it.
+ if (newRight == BooleanLiteral.TRUE || rightContext.containsUnEvaluableExpression) {
+ return newLeft;
+ }
+
+ // both does not contains non-partition slot.
+ return new And(newLeft, newRight);
+ }
+
+ @Override
+ public Expression visitOr(Or node, Context parentContext) {
+ // handle left node
+ Context leftContext = new Context();
+ Expression newLeft = node.left().accept(this, leftContext);
+ // handle right node
+ Context rightContext = new Context();
+ Expression newRight = node.right().accept(this, rightContext);
+
+ // if anyone of them is TRUE or contains non-partition slot, just return TRUE.
+ if (newLeft == BooleanLiteral.TRUE || newRight == BooleanLiteral.TRUE
+ || leftContext.containsUnEvaluableExpression || rightContext.containsUnEvaluableExpression) {
+ return BooleanLiteral.TRUE;
+ }
+
+ return new Or(newLeft, newRight);
+ }
+
+ /**
+ * Context
+ */
+ @VisibleForTesting
+ public static class Context {
+ private boolean containsUnEvaluableExpression;
+ }
+
+ /**
+ * The detector only indicate that whether a predicate contains interested slots or not,
+ * and do not change the predicate.
+ */
+ @VisibleForTesting
+ public static class ExpressionEvaluableDetector extends DefaultExpressionRewriter<Context> {
+ private final Set<Slot> partitionSlots;
+
+ public ExpressionEvaluableDetector(Set<Slot> partitionSlots) {
+ this.partitionSlots = Objects.requireNonNull(partitionSlots);
+ }
+
+ /**
+ * Return true if expression does NOT contains un-evaluable expression.
+ */
+ @VisibleForTesting
+ public boolean detect(Expression expression) {
+ Context context = new Context();
+ expression.accept(this, context);
+ return !context.containsUnEvaluableExpression;
+ }
+
+ @Override
+ public Expression visit(Expression originExpr, Context parentContext) {
+ if (parentContext.containsUnEvaluableExpression) {
+ return originExpr;
+ }
+
+ // todo: Should we consider un-deterministic functions as un-evaluable expression?
+ // if (originExpr instanceof Function) {
+ // Function function = (Function) originExpr;
+ // if (Env.getCurrentEnv().isNondeterministicFunction(function.getName())) {
+ // parentContext.containsUnEvaluableExpression = true;
+ // return originExpr;
+ // }
+ // }
Review Comment:
not need, some Nondeterministic functions are foldable and others are not. FoldConstantRuleOnFE can detect it.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruneExpressionExtractor.java:
##########
@@ -0,0 +1,204 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.rules.expression.rules.PartitionPruneExpressionExtractor.Context;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SubqueryExpr;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * PartitionPruneExpressionExtractor
+ *
+ * This rewriter only used to extract the expression that can be used in partition pruning from
+ * the whole predicate expression.
+ * The theory of extractor is pretty simple:
+ * A:Sort the expression in two kinds:
+ * 1. evaluable-expression (let's mark it as E).
+ * Expressions that can be evaluated in the partition pruning stage.
+ * In the other word: not contains non-partition slots or deterministic expression.
+ * 2. un-evaluable-expression (let's mark it as UE).
+ * Expressions that can NOT be evaluated in the partition pruning stage.
+ * In the other word: contains non-partition slots or deterministic expression.
+ *
+ * B: Travel the predicate, only point on AND and OR operator, following the rule:
+ * (E and UE) -> (E and TRUE) -> E
+ * (UE and UE) -> TRUE
+ * (E and E) -> (E and E)
+ * (E or UE) -> TRUE
+ * (UE or UE) -> TRUE
+ * (E or E) -> (E or E)
+ *
+ * e.g.
+ * (part = 1 and non_part = 'a') or (part = 2)
+ * -> (part = 1 and true) or (part = 2)
+ * -> (part = 1) or (part = 2)
+ *
+ * It's better that do some expression optimize(like fold, eliminate etc.) on predicate before this step.
+ */
+public class PartitionPruneExpressionExtractor extends DefaultExpressionRewriter<Context> {
+ private final ExpressionEvaluableDetector expressionEvaluableDetector;
+
+ private PartitionPruneExpressionExtractor(Set<Slot> interestedSlots, CascadesContext cascadesContext) {
+ this.expressionEvaluableDetector = new ExpressionEvaluableDetector(interestedSlots);
+ }
+
+ /**
+ * Extract partition prune expression from predicate
+ */
+ public static Expression extract(Expression predicate,
+ Set<Slot> partitionSlots,
+ CascadesContext cascadesContext) {
+ predicate = predicate.accept(FoldConstantRuleOnFE.VISITOR_INSTANCE, null);
+ PartitionPruneExpressionExtractor rewriter =
+ new PartitionPruneExpressionExtractor(partitionSlots, cascadesContext);
+ Context context = new Context();
+ Expression partitionPruneExpression = predicate.accept(rewriter, context);
+ if (context.containsUnEvaluableExpression) {
+ return BooleanLiteral.TRUE;
+ }
+ return partitionPruneExpression;
+ }
+
+ @Override
+ public Expression visit(Expression originExpr, Context parentContext) {
+ if (originExpr instanceof And) {
+ return this.visitAnd((And) originExpr, parentContext);
+ }
+ if (originExpr instanceof Or) {
+ return this.visitOr((Or) originExpr, parentContext);
+ }
+
+ return originExpr.accept(expressionEvaluableDetector, parentContext);
+ }
+
+ @Override
+ public Expression visitAnd(And node, Context parentContext) {
+ // handle left node
+ Context leftContext = new Context();
+ Expression newLeft = node.left().accept(this, leftContext);
+ // handle right node
+ Context rightContext = new Context();
+ Expression newRight = node.right().accept(this, rightContext);
+
+ // if anyone of them is FALSE, the whole expression should be FALSE.
+ if (newLeft == BooleanLiteral.FALSE || newRight == BooleanLiteral.FALSE) {
+ return BooleanLiteral.FALSE;
+ }
+
+ // If left node contains non-partition slot or is TURE, just discard it.
+ if (newLeft == BooleanLiteral.TRUE || leftContext.containsUnEvaluableExpression) {
+ return rightContext.containsUnEvaluableExpression ? BooleanLiteral.TRUE : newRight;
+ }
+
+ // If right node contains non-partition slot or is TURE, just discard it.
+ if (newRight == BooleanLiteral.TRUE || rightContext.containsUnEvaluableExpression) {
+ return newLeft;
+ }
+
+ // both does not contains non-partition slot.
+ return new And(newLeft, newRight);
+ }
+
+ @Override
+ public Expression visitOr(Or node, Context parentContext) {
+ // handle left node
+ Context leftContext = new Context();
+ Expression newLeft = node.left().accept(this, leftContext);
+ // handle right node
+ Context rightContext = new Context();
+ Expression newRight = node.right().accept(this, rightContext);
+
+ // if anyone of them is TRUE or contains non-partition slot, just return TRUE.
+ if (newLeft == BooleanLiteral.TRUE || newRight == BooleanLiteral.TRUE
+ || leftContext.containsUnEvaluableExpression || rightContext.containsUnEvaluableExpression) {
+ return BooleanLiteral.TRUE;
+ }
+
+ return new Or(newLeft, newRight);
+ }
+
+ /**
+ * Context
+ */
+ @VisibleForTesting
+ public static class Context {
+ private boolean containsUnEvaluableExpression;
+ }
+
+ /**
+ * The detector only indicate that whether a predicate contains interested slots or not,
+ * and do not change the predicate.
+ */
+ @VisibleForTesting
+ public static class ExpressionEvaluableDetector extends DefaultExpressionRewriter<Context> {
+ private final Set<Slot> partitionSlots;
+
+ public ExpressionEvaluableDetector(Set<Slot> partitionSlots) {
+ this.partitionSlots = Objects.requireNonNull(partitionSlots);
+ }
+
+ /**
+ * Return true if expression does NOT contains un-evaluable expression.
+ */
+ @VisibleForTesting
+ public boolean detect(Expression expression) {
+ Context context = new Context();
+ expression.accept(this, context);
+ return !context.containsUnEvaluableExpression;
+ }
+
+ @Override
+ public Expression visit(Expression originExpr, Context parentContext) {
+ if (parentContext.containsUnEvaluableExpression) {
+ return originExpr;
+ }
+
+ // todo: Should we consider un-deterministic functions as un-evaluable expression?
+ // if (originExpr instanceof Function) {
+ // Function function = (Function) originExpr;
+ // if (Env.getCurrentEnv().isNondeterministicFunction(function.getName())) {
+ // parentContext.containsUnEvaluableExpression = true;
+ // return originExpr;
+ // }
+ // }
+ if (originExpr instanceof SubqueryExpr) {
+ parentContext.containsUnEvaluableExpression = true;
+ return originExpr;
+ }
+
+ if (originExpr instanceof Slot) {
+ parentContext.containsUnEvaluableExpression = !partitionSlots.contains(originExpr);
+ return originExpr;
+ }
+
+ super.visit(originExpr, parentContext);
+ return originExpr;
+ }
+ }
+}
Review Comment:
You can refactor this class to be shorter
```java
public static class ExpressionEvaluableDetector {
public boolean detect(Expression expression, Set<Slot> partitionSlots) {
AtomicBoolean containsUnEvaluableExpression = new AtomicBoolean();
expression.foreach(expr -> {
if (containsUnEvaluableExpression.get())) {
// don't traverse children
return true;
}
if (expr instanceof SubqueryExpr) {
containsUnEvaluableExpression.set(true);
return true;
}
if (expr instanceof Slot && !partitionSlots.contains(expr)) {
containsUnEvaluableExpression.set(true);
return true;
}
// continue traverse children
return false;
});
return containsUnEvaluableExpression.get();
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] New expression extractor for partitions pruning. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2172178542
<details>
<summary>TPC-DS: <b>Total hot run time: 172407 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 91e523aa1074b21dfe00fcc978f6af9167fc70fb, data reload: false
query1 920 387 365 365
query2 6430 2350 2216 2216
query3 6634 209 209 209
query4 19178 17307 17302 17302
query5 3614 460 441 441
query6 236 158 161 158
query7 4613 297 289 289
query8 337 289 279 279
query9 8702 2375 2365 2365
query10 571 297 273 273
query11 10780 10205 10104 10104
query12 131 83 85 83
query13 1645 363 362 362
query14 9945 6724 6071 6071
query15 224 190 187 187
query16 7210 299 277 277
query17 1423 534 512 512
query18 1857 270 265 265
query19 195 155 158 155
query20 91 85 82 82
query21 209 132 127 127
query22 4375 4027 4077 4027
query23 33768 33614 33733 33614
query24 11217 2944 2891 2891
query25 617 363 400 363
query26 1177 161 164 161
query27 2474 336 333 333
query28 7106 2126 2108 2108
query29 894 661 608 608
query30 254 151 152 151
query31 968 759 780 759
query32 91 52 54 52
query33 746 282 277 277
query34 1066 494 487 487
query35 748 675 656 656
query36 1134 951 982 951
query37 172 72 77 72
query38 2929 2847 2842 2842
query39 894 843 839 839
query40 206 135 130 130
query41 58 52 55 52
query42 108 104 106 104
query43 585 543 532 532
query44 1178 721 731 721
query45 199 155 162 155
query46 1072 728 695 695
query47 1885 1799 1816 1799
query48 349 291 289 289
query49 849 384 394 384
query50 756 396 392 392
query51 6835 6610 6639 6610
query52 104 88 88 88
query53 350 287 289 287
query54 865 445 433 433
query55 76 71 74 71
query56 299 252 255 252
query57 1138 1036 1048 1036
query58 245 245 228 228
query59 3372 2957 2960 2957
query60 282 268 272 268
query61 90 93 88 88
query62 614 425 449 425
query63 311 293 283 283
query64 8834 2345 1839 1839
query65 3156 3120 3091 3091
query66 824 335 344 335
query67 15501 15033 14860 14860
query68 5239 551 564 551
query69 606 434 355 355
query70 1158 1146 1079 1079
query71 454 279 275 275
query72 8542 5569 5410 5410
query73 777 331 335 331
query74 5983 5591 5554 5554
query75 3668 2634 2670 2634
query76 3086 957 943 943
query77 614 291 292 291
query78 10454 9815 9952 9815
query79 5710 498 516 498
query80 1981 464 458 458
query81 597 212 227 212
query82 1539 102 101 101
query83 317 165 162 162
query84 279 80 91 80
query85 1554 275 263 263
query86 546 312 291 291
query87 3327 3108 3043 3043
query88 5151 2332 2324 2324
query89 484 376 379 376
query90 1757 190 186 186
query91 127 98 97 97
query92 58 50 47 47
query93 5605 509 508 508
query94 1174 192 182 182
query95 404 315 312 312
query96 595 263 266 263
query97 3233 3049 3021 3021
query98 209 203 197 197
query99 1313 856 831 831
Total cold run time: 282257 ms
Total hot run time: 172407 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173116080
<details>
<summary>ClickBench: <b>Total hot run time: 30.8 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b0a828dcb1cfd5b422c86356f5ee5b1c4d534a54, data reload: false
query1 0.04 0.03 0.03
query2 0.08 0.04 0.03
query3 0.22 0.06 0.05
query4 1.67 0.08 0.08
query5 0.50 0.48 0.48
query6 1.12 0.72 0.72
query7 0.02 0.01 0.01
query8 0.05 0.04 0.05
query9 0.54 0.48 0.49
query10 0.55 0.55 0.54
query11 0.16 0.11 0.11
query12 0.15 0.12 0.13
query13 0.59 0.58 0.58
query14 0.78 0.76 0.80
query15 0.86 0.81 0.81
query16 0.38 0.37 0.36
query17 1.04 1.02 0.98
query18 0.20 0.26 0.23
query19 1.77 1.67 1.72
query20 0.01 0.01 0.01
query21 15.42 0.64 0.64
query22 4.12 6.84 2.21
query23 18.32 1.38 1.32
query24 2.76 0.28 0.22
query25 0.20 0.09 0.08
query26 0.27 0.18 0.18
query27 0.08 0.08 0.08
query28 13.36 1.02 1.01
query29 12.66 3.29 3.30
query30 0.26 0.06 0.05
query31 2.87 0.39 0.39
query32 3.27 0.47 0.48
query33 2.81 2.86 2.93
query34 16.96 4.41 4.42
query35 4.46 4.45 4.48
query36 0.65 0.46 0.46
query37 0.18 0.16 0.15
query38 0.15 0.14 0.14
query39 0.04 0.04 0.03
query40 0.18 0.14 0.15
query41 0.10 0.05 0.05
query42 0.05 0.05 0.05
query43 0.05 0.04 0.03
Total cold run time: 109.95 s
Total hot run time: 30.8 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]
Posted by "XuPengfei-1020 (via GitHub)" <gi...@apache.org>.
XuPengfei-1020 commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2175004591
> could u add a case that can reproduce this problem?
The case that can reporduce the issue at above, "Proposed changes" chapter (Added by @924060929 ).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] New expression extractor for partitions pruning. [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #36326:
URL: https://github.com/apache/doris/pull/36326#discussion_r1642139159
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruneExpressionExtractor.java:
##########
@@ -0,0 +1,204 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.rules.expression.rules.PartitionPruneExpressionExtractor.Context;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SubqueryExpr;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * PartitionPruneExpressionExtractor
+ *
+ * This rewriter only used to extract the expression that can be used in partition pruning from
+ * the whole predicate expression.
+ * The theory of extractor is pretty simple:
+ * A:Sort the expression in two kinds:
+ * 1. evaluable-expression (let's mark it as E).
+ * Expressions that can be evaluated in the partition pruning stage.
+ * In the other word: not contains non-partition slots or deterministic expression.
+ * 2. un-evaluable-expression (let's mark it as UE).
+ * Expressions that can NOT be evaluated in the partition pruning stage.
+ * In the other word: contains non-partition slots or deterministic expression.
+ *
+ * B: Travel the predicate, only point on AND and OR operator, following the rule:
+ * (E and UE) -> (E and TRUE) -> E
+ * (UE and UE) -> TRUE
+ * (E and E) -> (E and E)
+ * (E or UE) -> TRUE
+ * (UE or UE) -> TRUE
+ * (E or E) -> (E or E)
+ *
+ * e.g.
+ * (part = 1 and non_part = 'a') or (part = 2)
+ * -> (part = 1 and true) or (part = 2)
+ * -> (part = 1) or (part = 2)
+ *
+ * It's better that do some expression optimize(like fold, eliminate etc.) on predicate before this step.
+ */
+public class PartitionPruneExpressionExtractor extends DefaultExpressionRewriter<Context> {
+ private final ExpressionEvaluableDetector expressionEvaluableDetector;
+
+ private PartitionPruneExpressionExtractor(Set<Slot> interestedSlots, CascadesContext cascadesContext) {
+ this.expressionEvaluableDetector = new ExpressionEvaluableDetector(interestedSlots);
+ }
+
+ /**
+ * Extract partition prune expression from predicate
+ */
+ public static Expression extract(Expression predicate,
+ Set<Slot> partitionSlots,
+ CascadesContext cascadesContext) {
+ predicate = predicate.accept(FoldConstantRuleOnFE.VISITOR_INSTANCE, null);
Review Comment:
new an ExpressionRewriteContext as context parameter:
```java
predicate = predicate.accept(FoldConstantRuleOnFE.VISITOR_INSTANCE, new ExpressionRewriteContext(cascadesContext));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173085290
<details>
<summary>TPC-H: <b>Total hot run time: 39594 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b0a828dcb1cfd5b422c86356f5ee5b1c4d534a54, data reload: false
------ Round 1 ----------------------------------
q1 17631 4332 4237 4237
q2 2029 190 198 190
q3 10427 1197 1107 1107
q4 10187 804 734 734
q5 7449 2635 2602 2602
q6 221 134 135 134
q7 960 610 595 595
q8 9214 2032 2072 2032
q9 8831 6456 6442 6442
q10 8905 3712 3766 3712
q11 452 231 240 231
q12 476 231 224 224
q13 17777 2981 2967 2967
q14 265 228 224 224
q15 505 475 475 475
q16 516 371 382 371
q17 949 659 677 659
q18 8058 7486 7378 7378
q19 2204 1488 1536 1488
q20 652 298 324 298
q21 4914 3162 3788 3162
q22 386 332 339 332
Total cold run time: 113008 ms
Total hot run time: 39594 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4335 4179 4259 4179
q2 373 273 274 273
q3 2945 2705 2707 2705
q4 2096 1732 1684 1684
q5 5441 5618 5488 5488
q6 227 127 131 127
q7 2180 1823 1813 1813
q8 3276 3419 3388 3388
q9 8685 8687 8735 8687
q10 4127 3860 3879 3860
q11 579 485 485 485
q12 774 578 599 578
q13 16150 3227 3184 3184
q14 298 274 291 274
q15 520 468 501 468
q16 504 440 446 440
q17 1784 1515 1491 1491
q18 8080 7829 7825 7825
q19 2831 1707 1637 1637
q20 2138 1886 1869 1869
q21 5034 4867 4890 4867
q22 638 593 557 557
Total cold run time: 73015 ms
Total hot run time: 55879 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173431850
<details>
<summary>ClickBench: <b>Total hot run time: 30.62 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0c072d3adaecd053725e993dce322a63de6a8e9d, data reload: false
query1 0.04 0.03 0.03
query2 0.08 0.04 0.04
query3 0.23 0.04 0.04
query4 1.69 0.07 0.07
query5 0.51 0.52 0.49
query6 1.14 0.74 0.72
query7 0.02 0.02 0.01
query8 0.05 0.04 0.04
query9 0.55 0.49 0.48
query10 0.54 0.54 0.54
query11 0.15 0.11 0.12
query12 0.15 0.12 0.12
query13 0.59 0.59 0.60
query14 0.79 0.79 0.77
query15 0.84 0.81 0.81
query16 0.36 0.37 0.36
query17 0.96 1.06 1.05
query18 0.22 0.24 0.25
query19 1.90 1.70 1.72
query20 0.02 0.01 0.01
query21 15.41 0.67 0.66
query22 4.28 6.96 2.00
query23 18.26 1.25 1.24
query24 2.18 0.22 0.21
query25 0.16 0.08 0.09
query26 0.26 0.18 0.18
query27 0.09 0.08 0.09
query28 13.17 1.02 1.00
query29 12.61 3.22 3.19
query30 0.26 0.06 0.06
query31 2.86 0.38 0.39
query32 3.29 0.49 0.47
query33 2.91 2.87 2.89
query34 17.18 4.46 4.42
query35 4.48 4.53 4.59
query36 0.66 0.48 0.46
query37 0.18 0.15 0.15
query38 0.15 0.14 0.15
query39 0.04 0.03 0.04
query40 0.17 0.13 0.13
query41 0.10 0.06 0.04
query42 0.06 0.05 0.04
query43 0.05 0.04 0.04
Total cold run time: 109.64 s
Total hot run time: 30.62 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173337884
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2174875101
could u add a case that can reproduce this problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173111157
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org