You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "XuPengfei-1020 (via GitHub)" <gi...@apache.org> on 2024/06/14 12:18:24 UTC

[PR] New expression extractor for partitions pruning. [doris]

XuPengfei-1020 opened a new pull request, #36326:
URL: https://github.com/apache/doris/pull/36326

   ## Proposed changes
   Original extractor(TryEliminateUninterestedPredicates.java) caused some errors while try to derive the expressions which can be used for pruning partitions.
   I tried to write a new extractor(and with unit tests) for pruning partitions, it is more simple and reliable (I think).
   
   The theory of extractor is pretty simple:
   A:Sort the expression in two kinds:
     1. evaluable-expression (let's mark it as E).
       Expressions that can be evaluated in the partition pruning stage.
       In the other word: not contains non-partition slots or deterministic expression.
     2. un-evaluable-expression (let's mark it as UE).
       Expressions that can NOT be evaluated in the partition pruning stage.
       In the other word: contains non-partition slots or deterministic expression.
   
   B: Travel the predicate, only point on AND and OR operator, following the rule:
     (E and UE) -> (E and TRUE) -> E
     (UE and UE) -> TRUE
     (E and E) -> (E and E)
     (E or UE) -> TRUE
     (UE or UE) -> TRUE
     (E or E) -> (E or E)
   <!--Describe your changes.-->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] New expression extractor for partitions pruning. [doris]

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2172083476

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]

Posted by "XuPengfei-1020 (via GitHub)" <gi...@apache.org>.
XuPengfei-1020 commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2172987456

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] New expression extractor for partitions pruning. [doris]

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #36326:
URL: https://github.com/apache/doris/pull/36326#discussion_r1642139159


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruneExpressionExtractor.java:
##########
@@ -0,0 +1,204 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.rules.expression.rules.PartitionPruneExpressionExtractor.Context;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SubqueryExpr;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * PartitionPruneExpressionExtractor
+ *
+ * This rewriter only used to extract the expression that can be used in partition pruning from
+ *     the whole predicate expression.
+ * The theory of extractor is pretty simple:
+ * A:Sort the expression in two kinds:
+ *  1. evaluable-expression (let's mark it as E).
+ *      Expressions that can be evaluated in the partition pruning stage.
+ *      In the other word: not contains non-partition slots or deterministic expression.
+ *  2. un-evaluable-expression (let's mark it as UE).
+ *      Expressions that can NOT be evaluated in the partition pruning stage.
+ *      In the other word: contains non-partition slots or deterministic expression.
+ *
+ * B: Travel the predicate, only point on AND and OR operator, following the rule:
+ *    (E and UE) -> (E and TRUE) -> E
+ *    (UE and UE) -> TRUE
+ *    (E and E) -> (E and E)
+ *    (E or UE) -> TRUE
+ *    (UE or UE) -> TRUE
+ *    (E or E) -> (E or E)
+ *
+ * e.g.
+ *    (part = 1 and non_part = 'a') or (part = 2)
+ * -> (part = 1 and true) or (part = 2)
+ * -> (part = 1) or (part = 2)
+ *
+ * It's better that do some expression optimize(like fold, eliminate etc.) on predicate before this step.
+ */
+public class PartitionPruneExpressionExtractor extends DefaultExpressionRewriter<Context> {
+    private final ExpressionEvaluableDetector expressionEvaluableDetector;
+
+    private PartitionPruneExpressionExtractor(Set<Slot> interestedSlots, CascadesContext cascadesContext) {
+        this.expressionEvaluableDetector = new ExpressionEvaluableDetector(interestedSlots);
+    }
+
+    /**
+     * Extract partition prune expression from predicate
+     */
+    public static Expression extract(Expression predicate,
+                                     Set<Slot> partitionSlots,
+                                     CascadesContext cascadesContext) {
+        predicate = predicate.accept(FoldConstantRuleOnFE.VISITOR_INSTANCE, null);

Review Comment:
   new an ExpressionRewriteContext as context parameter



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruneExpressionExtractor.java:
##########
@@ -0,0 +1,204 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.rules.expression.rules.PartitionPruneExpressionExtractor.Context;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SubqueryExpr;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * PartitionPruneExpressionExtractor
+ *
+ * This rewriter only used to extract the expression that can be used in partition pruning from
+ *     the whole predicate expression.
+ * The theory of extractor is pretty simple:
+ * A:Sort the expression in two kinds:
+ *  1. evaluable-expression (let's mark it as E).
+ *      Expressions that can be evaluated in the partition pruning stage.
+ *      In the other word: not contains non-partition slots or deterministic expression.
+ *  2. un-evaluable-expression (let's mark it as UE).
+ *      Expressions that can NOT be evaluated in the partition pruning stage.
+ *      In the other word: contains non-partition slots or deterministic expression.
+ *
+ * B: Travel the predicate, only point on AND and OR operator, following the rule:
+ *    (E and UE) -> (E and TRUE) -> E
+ *    (UE and UE) -> TRUE
+ *    (E and E) -> (E and E)
+ *    (E or UE) -> TRUE
+ *    (UE or UE) -> TRUE
+ *    (E or E) -> (E or E)
+ *
+ * e.g.
+ *    (part = 1 and non_part = 'a') or (part = 2)
+ * -> (part = 1 and true) or (part = 2)
+ * -> (part = 1) or (part = 2)
+ *
+ * It's better that do some expression optimize(like fold, eliminate etc.) on predicate before this step.
+ */
+public class PartitionPruneExpressionExtractor extends DefaultExpressionRewriter<Context> {
+    private final ExpressionEvaluableDetector expressionEvaluableDetector;
+
+    private PartitionPruneExpressionExtractor(Set<Slot> interestedSlots, CascadesContext cascadesContext) {
+        this.expressionEvaluableDetector = new ExpressionEvaluableDetector(interestedSlots);
+    }
+
+    /**
+     * Extract partition prune expression from predicate
+     */
+    public static Expression extract(Expression predicate,
+                                     Set<Slot> partitionSlots,
+                                     CascadesContext cascadesContext) {
+        predicate = predicate.accept(FoldConstantRuleOnFE.VISITOR_INSTANCE, null);
+        PartitionPruneExpressionExtractor rewriter =
+                new PartitionPruneExpressionExtractor(partitionSlots, cascadesContext);
+        Context context = new Context();
+        Expression partitionPruneExpression = predicate.accept(rewriter, context);
+        if (context.containsUnEvaluableExpression) {
+            return BooleanLiteral.TRUE;
+        }
+        return partitionPruneExpression;
+    }
+
+    @Override
+    public Expression visit(Expression originExpr, Context parentContext) {
+        if (originExpr instanceof And) {
+            return this.visitAnd((And) originExpr, parentContext);
+        }
+        if (originExpr instanceof Or) {
+            return this.visitOr((Or) originExpr, parentContext);
+        }
+
+        return originExpr.accept(expressionEvaluableDetector, parentContext);
+    }
+
+    @Override
+    public Expression visitAnd(And node, Context parentContext) {
+        // handle left node
+        Context leftContext = new Context();
+        Expression newLeft = node.left().accept(this, leftContext);
+        // handle right node
+        Context rightContext = new Context();
+        Expression newRight = node.right().accept(this, rightContext);
+
+        // if anyone of them is FALSE, the whole expression should be FALSE.
+        if (newLeft == BooleanLiteral.FALSE || newRight == BooleanLiteral.FALSE) {
+            return BooleanLiteral.FALSE;
+        }
+
+        // If left node contains non-partition slot or is TURE, just discard it.
+        if (newLeft == BooleanLiteral.TRUE || leftContext.containsUnEvaluableExpression) {
+            return rightContext.containsUnEvaluableExpression ? BooleanLiteral.TRUE : newRight;
+        }
+
+        // If right node contains non-partition slot or is TURE, just discard it.
+        if (newRight == BooleanLiteral.TRUE || rightContext.containsUnEvaluableExpression) {
+            return newLeft;
+        }
+
+        // both does not contains non-partition slot.
+        return new And(newLeft, newRight);
+    }
+
+    @Override
+    public Expression visitOr(Or node, Context parentContext) {
+        // handle left node
+        Context leftContext = new Context();
+        Expression newLeft = node.left().accept(this, leftContext);
+        // handle right node
+        Context rightContext = new Context();
+        Expression newRight = node.right().accept(this, rightContext);
+
+        // if anyone of them is TRUE or contains non-partition slot, just return TRUE.
+        if (newLeft == BooleanLiteral.TRUE || newRight == BooleanLiteral.TRUE
+                || leftContext.containsUnEvaluableExpression || rightContext.containsUnEvaluableExpression) {
+            return BooleanLiteral.TRUE;
+        }
+
+        return new Or(newLeft, newRight);
+    }
+
+    /**
+     * Context
+     */
+    @VisibleForTesting
+    public static class Context {
+        private boolean containsUnEvaluableExpression;
+    }
+
+    /**
+     * The detector only indicate that whether a predicate contains interested slots or not,
+     * and do not change the predicate.
+     */
+    @VisibleForTesting
+    public static class ExpressionEvaluableDetector extends DefaultExpressionRewriter<Context> {
+        private final Set<Slot> partitionSlots;
+
+        public ExpressionEvaluableDetector(Set<Slot> partitionSlots) {
+            this.partitionSlots = Objects.requireNonNull(partitionSlots);

Review Comment:
   and a simple exception message here, e.g.
   ```java
   this.partitionSlots = Objects.requireNonNull(partitionSlots, "partitionSlots can not be null);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173104650

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 174163 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit b0a828dcb1cfd5b422c86356f5ee5b1c4d534a54, data reload: false
   
   query1	924	383	383	383
   query2	6455	2368	2323	2323
   query3	6638	205	209	205
   query4	22524	17413	17476	17413
   query5	3584	478	470	470
   query6	244	160	167	160
   query7	4607	304	286	286
   query8	319	298	282	282
   query9	8451	2392	2393	2392
   query10	556	299	288	288
   query11	10536	10147	9935	9935
   query12	116	90	84	84
   query13	1629	362	362	362
   query14	10031	7596	7740	7596
   query15	255	200	207	200
   query16	7779	272	284	272
   query17	1564	542	543	542
   query18	2020	295	286	286
   query19	208	175	167	167
   query20	93	85	83	83
   query21	206	129	143	129
   query22	4647	4424	4374	4374
   query23	34031	33649	33551	33551
   query24	10161	2954	2920	2920
   query25	572	353	353	353
   query26	701	149	165	149
   query27	2191	310	320	310
   query28	5824	2071	2046	2046
   query29	854	618	610	610
   query30	234	152	149	149
   query31	950	719	737	719
   query32	90	50	55	50
   query33	640	279	269	269
   query34	845	472	466	466
   query35	706	607	621	607
   query36	1057	964	958	958
   query37	140	67	70	67
   query38	2870	2727	2746	2727
   query39	852	802	860	802
   query40	206	126	121	121
   query41	54	49	49	49
   query42	115	99	95	95
   query43	600	542	547	542
   query44	1083	727	722	722
   query45	191	160	162	160
   query46	1074	719	711	711
   query47	1885	1767	1756	1756
   query48	363	293	305	293
   query49	841	395	417	395
   query50	760	396	395	395
   query51	6787	6729	6703	6703
   query52	98	96	91	91
   query53	361	287	296	287
   query54	851	446	430	430
   query55	73	75	75	75
   query56	291	258	263	258
   query57	1131	1051	1067	1051
   query58	251	248	238	238
   query59	3372	3373	3156	3156
   query60	289	256	284	256
   query61	96	90	92	90
   query62	590	451	454	451
   query63	313	286	289	286
   query64	8452	2296	1752	1752
   query65	3159	3081	3105	3081
   query66	734	328	327	327
   query67	15142	15366	14873	14873
   query68	4548	558	538	538
   query69	458	316	305	305
   query70	1173	1028	1137	1028
   query71	371	279	271	271
   query72	6977	5519	5477	5477
   query73	740	327	323	323
   query74	5928	5515	5392	5392
   query75	3363	2638	2706	2638
   query76	2184	943	913	913
   query77	409	304	304	304
   query78	10651	9857	9716	9716
   query79	2329	521	519	519
   query80	1129	529	455	455
   query81	567	223	222	222
   query82	740	102	107	102
   query83	224	169	178	169
   query84	237	88	90	88
   query85	1377	290	271	271
   query86	487	305	291	291
   query87	3291	3099	3078	3078
   query88	4344	2362	2358	2358
   query89	465	374	387	374
   query90	1704	200	195	195
   query91	128	101	101	101
   query92	58	50	50	50
   query93	1941	509	502	502
   query94	1119	193	186	186
   query95	398	316	314	314
   query96	580	269	264	264
   query97	3261	3052	3058	3052
   query98	218	199	188	188
   query99	1204	871	850	850
   Total cold run time: 267869 ms
   Total hot run time: 174163 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] New expression extractor for partitions pruning. [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2172152606

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 39833 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 91e523aa1074b21dfe00fcc978f6af9167fc70fb, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17616	4401	4314	4314
   q2	2015	194	193	193
   q3	10457	1091	1172	1091
   q4	10198	834	793	793
   q5	7482	2616	2659	2616
   q6	226	137	137	137
   q7	953	599	590	590
   q8	9227	2085	2090	2085
   q9	8978	6516	6495	6495
   q10	9011	3688	3682	3682
   q11	479	241	240	240
   q12	452	223	224	223
   q13	18785	2971	2990	2971
   q14	260	218	223	218
   q15	518	476	473	473
   q16	523	368	378	368
   q17	975	664	692	664
   q18	8014	7558	7350	7350
   q19	7458	1509	1465	1465
   q20	665	323	336	323
   q21	4885	3212	4049	3212
   q22	401	330	345	330
   Total cold run time: 119578 ms
   Total hot run time: 39833 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4445	4255	4232	4232
   q2	362	273	263	263
   q3	3075	2916	2859	2859
   q4	1985	1794	1759	1759
   q5	5524	5476	5422	5422
   q6	230	140	141	140
   q7	2228	1841	1823	1823
   q8	3288	3427	3432	3427
   q9	8655	8789	8721	8721
   q10	4192	3712	3766	3712
   q11	609	478	496	478
   q12	783	659	646	646
   q13	16995	3185	3153	3153
   q14	310	272	264	264
   q15	541	489	481	481
   q16	491	428	426	426
   q17	1812	1492	1477	1477
   q18	8163	7957	7764	7764
   q19	1859	1664	1460	1460
   q20	2197	1873	1858	1858
   q21	9146	4912	4740	4740
   q22	662	559	552	552
   Total cold run time: 77552 ms
   Total hot run time: 55657 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173394738

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 39849 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 0c072d3adaecd053725e993dce322a63de6a8e9d, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17632	4358	4257	4257
   q2	2043	184	190	184
   q3	10475	1141	1146	1141
   q4	10190	855	819	819
   q5	7467	2662	2653	2653
   q6	226	136	135	135
   q7	963	616	601	601
   q8	9287	2044	2089	2044
   q9	9451	6489	6467	6467
   q10	9052	3779	3745	3745
   q11	437	234	239	234
   q12	437	238	231	231
   q13	17800	2984	2991	2984
   q14	266	212	212	212
   q15	512	473	471	471
   q16	513	392	377	377
   q17	963	727	655	655
   q18	8033	7405	7469	7405
   q19	5334	1400	1447	1400
   q20	644	321	325	321
   q21	4890	3181	3989	3181
   q22	394	337	332	332
   Total cold run time: 117009 ms
   Total hot run time: 39849 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4363	4226	4217	4217
   q2	375	263	251	251
   q3	2987	2917	2896	2896
   q4	2004	1756	1749	1749
   q5	5604	5521	5494	5494
   q6	224	138	140	138
   q7	2198	1948	1870	1870
   q8	3303	3456	3401	3401
   q9	8774	8744	8767	8744
   q10	4169	3834	3835	3834
   q11	604	507	488	488
   q12	808	675	657	657
   q13	15945	3176	3188	3176
   q14	302	261	286	261
   q15	530	477	489	477
   q16	488	433	433	433
   q17	1821	1504	1489	1489
   q18	8179	7881	7880	7880
   q19	1787	1562	1651	1562
   q20	2151	1909	1880	1880
   q21	8089	5029	4760	4760
   q22	641	586	547	547
   Total cold run time: 75346 ms
   Total hot run time: 56204 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173111095

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] New expression extractor for partitions pruning. [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2172182669

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.49 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 91e523aa1074b21dfe00fcc978f6af9167fc70fb, data reload: false
   
   query1	0.04	0.03	0.03
   query2	0.08	0.04	0.04
   query3	0.23	0.05	0.05
   query4	1.67	0.06	0.06
   query5	0.49	0.48	0.47
   query6	1.14	0.72	0.72
   query7	0.02	0.01	0.02
   query8	0.05	0.04	0.05
   query9	0.54	0.49	0.48
   query10	0.54	0.52	0.54
   query11	0.15	0.12	0.12
   query12	0.14	0.12	0.12
   query13	0.60	0.58	0.60
   query14	0.76	0.80	0.78
   query15	0.83	0.81	0.81
   query16	0.36	0.35	0.36
   query17	0.94	0.98	1.00
   query18	0.22	0.24	0.25
   query19	1.82	1.67	1.84
   query20	0.01	0.02	0.02
   query21	15.41	0.65	0.64
   query22	4.32	6.66	2.14
   query23	18.29	1.33	1.18
   query24	2.24	0.21	0.21
   query25	0.15	0.07	0.10
   query26	0.29	0.18	0.17
   query27	0.07	0.08	0.08
   query28	13.20	1.01	0.99
   query29	12.58	3.31	3.25
   query30	0.25	0.07	0.06
   query31	2.85	0.40	0.39
   query32	3.26	0.49	0.47
   query33	2.87	2.88	2.92
   query34	16.92	4.44	4.39
   query35	4.46	4.43	4.47
   query36	0.65	0.47	0.46
   query37	0.18	0.16	0.15
   query38	0.15	0.14	0.15
   query39	0.04	0.04	0.04
   query40	0.16	0.14	0.14
   query41	0.10	0.05	0.04
   query42	0.06	0.05	0.04
   query43	0.05	0.04	0.04
   Total cold run time: 109.18 s
   Total hot run time: 30.49 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173419624

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 171456 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 0c072d3adaecd053725e993dce322a63de6a8e9d, data reload: false
   
   query1	923	388	382	382
   query2	7415	2352	2247	2247
   query3	6647	214	208	208
   query4	19032	17378	17141	17141
   query5	3549	473	476	473
   query6	237	180	154	154
   query7	4576	293	294	293
   query8	314	262	266	262
   query9	8507	2391	2392	2391
   query10	580	296	277	277
   query11	10468	10011	10011	10011
   query12	135	82	87	82
   query13	1630	373	372	372
   query14	10016	7607	7663	7607
   query15	231	185	191	185
   query16	7729	265	265	265
   query17	1897	559	509	509
   query18	1939	270	272	270
   query19	202	160	160	160
   query20	94	92	83	83
   query21	215	134	132	132
   query22	4345	4042	3995	3995
   query23	33899	33653	34046	33653
   query24	10715	2993	2976	2976
   query25	582	395	385	385
   query26	715	158	162	158
   query27	2308	333	324	324
   query28	5894	2069	2092	2069
   query29	880	623	634	623
   query30	235	154	156	154
   query31	971	778	759	759
   query32	93	52	58	52
   query33	670	290	285	285
   query34	885	489	473	473
   query35	772	648	630	630
   query36	1124	957	992	957
   query37	146	73	73	73
   query38	2997	2817	2839	2817
   query39	912	833	867	833
   query40	217	130	131	130
   query41	55	51	53	51
   query42	117	105	104	104
   query43	590	549	521	521
   query44	1082	716	726	716
   query45	196	161	161	161
   query46	1080	723	716	716
   query47	1834	1759	1762	1759
   query48	389	292	294	292
   query49	827	432	410	410
   query50	765	390	391	390
   query51	6870	6730	6609	6609
   query52	98	87	85	85
   query53	360	293	292	292
   query54	870	449	448	448
   query55	73	73	74	73
   query56	270	268	253	253
   query57	1119	1035	1051	1035
   query58	249	237	250	237
   query59	3371	3026	3011	3011
   query60	294	271	279	271
   query61	106	90	96	90
   query62	635	457	440	440
   query63	323	286	295	286
   query64	8612	2400	1766	1766
   query65	3201	3102	3098	3098
   query66	745	330	327	327
   query67	15433	15073	15046	15046
   query68	4511	537	549	537
   query69	569	461	379	379
   query70	1106	1157	1070	1070
   query71	441	271	279	271
   query72	7306	5786	2740	2740
   query73	752	330	332	330
   query74	5901	5497	5550	5497
   query75	3382	2664	2689	2664
   query76	2683	982	927	927
   query77	596	310	298	298
   query78	10375	9917	9809	9809
   query79	2780	514	509	509
   query80	2065	469	457	457
   query81	597	216	226	216
   query82	1001	116	107	107
   query83	304	169	163	163
   query84	262	91	83	83
   query85	1047	279	279	279
   query86	452	308	336	308
   query87	3298	3118	3100	3100
   query88	3704	2349	2335	2335
   query89	469	374	391	374
   query90	1596	194	193	193
   query91	129	103	99	99
   query92	60	47	50	47
   query93	1951	493	496	493
   query94	1067	189	184	184
   query95	403	304	312	304
   query96	589	267	266	266
   query97	3278	3095	3046	3046
   query98	226	206	195	195
   query99	1350	856	839	839
   Total cold run time: 268510 ms
   Total hot run time: 171456 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]

Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #36326:
URL: https://github.com/apache/doris/pull/36326


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] New expression extractor for partitions pruning. [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2167901580

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
   See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] New expression extractor for partitions pruning. [doris]

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #36326:
URL: https://github.com/apache/doris/pull/36326#discussion_r1642134171


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruneExpressionExtractor.java:
##########
@@ -0,0 +1,204 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.rules.expression.rules.PartitionPruneExpressionExtractor.Context;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SubqueryExpr;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * PartitionPruneExpressionExtractor
+ *
+ * This rewriter only used to extract the expression that can be used in partition pruning from
+ *     the whole predicate expression.
+ * The theory of extractor is pretty simple:
+ * A:Sort the expression in two kinds:
+ *  1. evaluable-expression (let's mark it as E).
+ *      Expressions that can be evaluated in the partition pruning stage.
+ *      In the other word: not contains non-partition slots or deterministic expression.
+ *  2. un-evaluable-expression (let's mark it as UE).
+ *      Expressions that can NOT be evaluated in the partition pruning stage.
+ *      In the other word: contains non-partition slots or deterministic expression.
+ *
+ * B: Travel the predicate, only point on AND and OR operator, following the rule:
+ *    (E and UE) -> (E and TRUE) -> E
+ *    (UE and UE) -> TRUE
+ *    (E and E) -> (E and E)
+ *    (E or UE) -> TRUE
+ *    (UE or UE) -> TRUE
+ *    (E or E) -> (E or E)
+ *
+ * e.g.
+ *    (part = 1 and non_part = 'a') or (part = 2)
+ * -> (part = 1 and true) or (part = 2)
+ * -> (part = 1) or (part = 2)
+ *
+ * It's better that do some expression optimize(like fold, eliminate etc.) on predicate before this step.
+ */
+public class PartitionPruneExpressionExtractor extends DefaultExpressionRewriter<Context> {
+    private final ExpressionEvaluableDetector expressionEvaluableDetector;
+
+    private PartitionPruneExpressionExtractor(Set<Slot> interestedSlots, CascadesContext cascadesContext) {
+        this.expressionEvaluableDetector = new ExpressionEvaluableDetector(interestedSlots);
+    }
+
+    /**
+     * Extract partition prune expression from predicate
+     */
+    public static Expression extract(Expression predicate,
+                                     Set<Slot> partitionSlots,
+                                     CascadesContext cascadesContext) {
+        predicate = predicate.accept(FoldConstantRuleOnFE.VISITOR_INSTANCE, null);
+        PartitionPruneExpressionExtractor rewriter =
+                new PartitionPruneExpressionExtractor(partitionSlots, cascadesContext);
+        Context context = new Context();
+        Expression partitionPruneExpression = predicate.accept(rewriter, context);
+        if (context.containsUnEvaluableExpression) {
+            return BooleanLiteral.TRUE;
+        }
+        return partitionPruneExpression;
+    }
+
+    @Override
+    public Expression visit(Expression originExpr, Context parentContext) {
+        if (originExpr instanceof And) {
+            return this.visitAnd((And) originExpr, parentContext);
+        }
+        if (originExpr instanceof Or) {
+            return this.visitOr((Or) originExpr, parentContext);
+        }
+
+        return originExpr.accept(expressionEvaluableDetector, parentContext);
+    }
+
+    @Override
+    public Expression visitAnd(And node, Context parentContext) {
+        // handle left node
+        Context leftContext = new Context();
+        Expression newLeft = node.left().accept(this, leftContext);
+        // handle right node
+        Context rightContext = new Context();
+        Expression newRight = node.right().accept(this, rightContext);
+
+        // if anyone of them is FALSE, the whole expression should be FALSE.
+        if (newLeft == BooleanLiteral.FALSE || newRight == BooleanLiteral.FALSE) {
+            return BooleanLiteral.FALSE;
+        }
+
+        // If left node contains non-partition slot or is TURE, just discard it.
+        if (newLeft == BooleanLiteral.TRUE || leftContext.containsUnEvaluableExpression) {
+            return rightContext.containsUnEvaluableExpression ? BooleanLiteral.TRUE : newRight;
+        }
+
+        // If right node contains non-partition slot or is TURE, just discard it.
+        if (newRight == BooleanLiteral.TRUE || rightContext.containsUnEvaluableExpression) {
+            return newLeft;
+        }
+
+        // both does not contains non-partition slot.
+        return new And(newLeft, newRight);
+    }
+
+    @Override
+    public Expression visitOr(Or node, Context parentContext) {
+        // handle left node
+        Context leftContext = new Context();
+        Expression newLeft = node.left().accept(this, leftContext);
+        // handle right node
+        Context rightContext = new Context();
+        Expression newRight = node.right().accept(this, rightContext);
+
+        // if anyone of them is TRUE or contains non-partition slot, just return TRUE.
+        if (newLeft == BooleanLiteral.TRUE || newRight == BooleanLiteral.TRUE
+                || leftContext.containsUnEvaluableExpression || rightContext.containsUnEvaluableExpression) {
+            return BooleanLiteral.TRUE;
+        }
+
+        return new Or(newLeft, newRight);
+    }
+
+    /**
+     * Context
+     */
+    @VisibleForTesting
+    public static class Context {
+        private boolean containsUnEvaluableExpression;
+    }
+
+    /**
+     * The detector only indicate that whether a predicate contains interested slots or not,
+     * and do not change the predicate.
+     */
+    @VisibleForTesting
+    public static class ExpressionEvaluableDetector extends DefaultExpressionRewriter<Context> {
+        private final Set<Slot> partitionSlots;
+
+        public ExpressionEvaluableDetector(Set<Slot> partitionSlots) {
+            this.partitionSlots = Objects.requireNonNull(partitionSlots);
+        }
+
+        /**
+         * Return true if expression does NOT contains un-evaluable expression.
+         */
+        @VisibleForTesting
+        public boolean detect(Expression expression) {
+            Context context = new Context();
+            expression.accept(this, context);
+            return !context.containsUnEvaluableExpression;
+        }
+
+        @Override
+        public Expression visit(Expression originExpr, Context parentContext) {
+            if (parentContext.containsUnEvaluableExpression) {
+                return originExpr;
+            }
+
+            // todo: Should we consider un-deterministic functions as un-evaluable expression?
+            // if (originExpr instanceof Function) {
+            //     Function function = (Function) originExpr;
+            //     if (Env.getCurrentEnv().isNondeterministicFunction(function.getName())) {
+            //         parentContext.containsUnEvaluableExpression = true;
+            //         return originExpr;
+            //     }
+            // }

Review Comment:
   not need, some Nondeterministic functions are foldable and others are not. FoldConstantRuleOnFE can detect it. 



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruneExpressionExtractor.java:
##########
@@ -0,0 +1,204 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.rules.expression.rules.PartitionPruneExpressionExtractor.Context;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SubqueryExpr;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * PartitionPruneExpressionExtractor
+ *
+ * This rewriter only used to extract the expression that can be used in partition pruning from
+ *     the whole predicate expression.
+ * The theory of extractor is pretty simple:
+ * A:Sort the expression in two kinds:
+ *  1. evaluable-expression (let's mark it as E).
+ *      Expressions that can be evaluated in the partition pruning stage.
+ *      In the other word: not contains non-partition slots or deterministic expression.
+ *  2. un-evaluable-expression (let's mark it as UE).
+ *      Expressions that can NOT be evaluated in the partition pruning stage.
+ *      In the other word: contains non-partition slots or deterministic expression.
+ *
+ * B: Travel the predicate, only point on AND and OR operator, following the rule:
+ *    (E and UE) -> (E and TRUE) -> E
+ *    (UE and UE) -> TRUE
+ *    (E and E) -> (E and E)
+ *    (E or UE) -> TRUE
+ *    (UE or UE) -> TRUE
+ *    (E or E) -> (E or E)
+ *
+ * e.g.
+ *    (part = 1 and non_part = 'a') or (part = 2)
+ * -> (part = 1 and true) or (part = 2)
+ * -> (part = 1) or (part = 2)
+ *
+ * It's better that do some expression optimize(like fold, eliminate etc.) on predicate before this step.
+ */
+public class PartitionPruneExpressionExtractor extends DefaultExpressionRewriter<Context> {
+    private final ExpressionEvaluableDetector expressionEvaluableDetector;
+
+    private PartitionPruneExpressionExtractor(Set<Slot> interestedSlots, CascadesContext cascadesContext) {
+        this.expressionEvaluableDetector = new ExpressionEvaluableDetector(interestedSlots);
+    }
+
+    /**
+     * Extract partition prune expression from predicate
+     */
+    public static Expression extract(Expression predicate,
+                                     Set<Slot> partitionSlots,
+                                     CascadesContext cascadesContext) {
+        predicate = predicate.accept(FoldConstantRuleOnFE.VISITOR_INSTANCE, null);
+        PartitionPruneExpressionExtractor rewriter =
+                new PartitionPruneExpressionExtractor(partitionSlots, cascadesContext);
+        Context context = new Context();
+        Expression partitionPruneExpression = predicate.accept(rewriter, context);
+        if (context.containsUnEvaluableExpression) {
+            return BooleanLiteral.TRUE;
+        }
+        return partitionPruneExpression;
+    }
+
+    @Override
+    public Expression visit(Expression originExpr, Context parentContext) {
+        if (originExpr instanceof And) {
+            return this.visitAnd((And) originExpr, parentContext);
+        }
+        if (originExpr instanceof Or) {
+            return this.visitOr((Or) originExpr, parentContext);
+        }
+
+        return originExpr.accept(expressionEvaluableDetector, parentContext);
+    }
+
+    @Override
+    public Expression visitAnd(And node, Context parentContext) {
+        // handle left node
+        Context leftContext = new Context();
+        Expression newLeft = node.left().accept(this, leftContext);
+        // handle right node
+        Context rightContext = new Context();
+        Expression newRight = node.right().accept(this, rightContext);
+
+        // if anyone of them is FALSE, the whole expression should be FALSE.
+        if (newLeft == BooleanLiteral.FALSE || newRight == BooleanLiteral.FALSE) {
+            return BooleanLiteral.FALSE;
+        }
+
+        // If left node contains non-partition slot or is TURE, just discard it.
+        if (newLeft == BooleanLiteral.TRUE || leftContext.containsUnEvaluableExpression) {
+            return rightContext.containsUnEvaluableExpression ? BooleanLiteral.TRUE : newRight;
+        }
+
+        // If right node contains non-partition slot or is TURE, just discard it.
+        if (newRight == BooleanLiteral.TRUE || rightContext.containsUnEvaluableExpression) {
+            return newLeft;
+        }
+
+        // both does not contains non-partition slot.
+        return new And(newLeft, newRight);
+    }
+
+    @Override
+    public Expression visitOr(Or node, Context parentContext) {
+        // handle left node
+        Context leftContext = new Context();
+        Expression newLeft = node.left().accept(this, leftContext);
+        // handle right node
+        Context rightContext = new Context();
+        Expression newRight = node.right().accept(this, rightContext);
+
+        // if anyone of them is TRUE or contains non-partition slot, just return TRUE.
+        if (newLeft == BooleanLiteral.TRUE || newRight == BooleanLiteral.TRUE
+                || leftContext.containsUnEvaluableExpression || rightContext.containsUnEvaluableExpression) {
+            return BooleanLiteral.TRUE;
+        }
+
+        return new Or(newLeft, newRight);
+    }
+
+    /**
+     * Context
+     */
+    @VisibleForTesting
+    public static class Context {
+        private boolean containsUnEvaluableExpression;
+    }
+
+    /**
+     * The detector only indicate that whether a predicate contains interested slots or not,
+     * and do not change the predicate.
+     */
+    @VisibleForTesting
+    public static class ExpressionEvaluableDetector extends DefaultExpressionRewriter<Context> {
+        private final Set<Slot> partitionSlots;
+
+        public ExpressionEvaluableDetector(Set<Slot> partitionSlots) {
+            this.partitionSlots = Objects.requireNonNull(partitionSlots);
+        }
+
+        /**
+         * Return true if expression does NOT contains un-evaluable expression.
+         */
+        @VisibleForTesting
+        public boolean detect(Expression expression) {
+            Context context = new Context();
+            expression.accept(this, context);
+            return !context.containsUnEvaluableExpression;
+        }
+
+        @Override
+        public Expression visit(Expression originExpr, Context parentContext) {
+            if (parentContext.containsUnEvaluableExpression) {
+                return originExpr;
+            }
+
+            // todo: Should we consider un-deterministic functions as un-evaluable expression?
+            // if (originExpr instanceof Function) {
+            //     Function function = (Function) originExpr;
+            //     if (Env.getCurrentEnv().isNondeterministicFunction(function.getName())) {
+            //         parentContext.containsUnEvaluableExpression = true;
+            //         return originExpr;
+            //     }
+            // }
+            if (originExpr instanceof SubqueryExpr) {
+                parentContext.containsUnEvaluableExpression = true;
+                return originExpr;
+            }
+
+            if (originExpr instanceof Slot) {
+                parentContext.containsUnEvaluableExpression = !partitionSlots.contains(originExpr);
+                return originExpr;
+            }
+
+            super.visit(originExpr, parentContext);
+            return originExpr;
+        }
+    }
+}

Review Comment:
   You can refactor this class to be shorter
   ```java
   public static class ExpressionEvaluableDetector {
       public boolean detect(Expression expression, Set<Slot> partitionSlots) {
           AtomicBoolean containsUnEvaluableExpression = new AtomicBoolean();
           expression.foreach(expr -> {
               if (containsUnEvaluableExpression.get())) {
                   // don't traverse children
                   return true;
               }
               if (expr instanceof SubqueryExpr) {
                   containsUnEvaluableExpression.set(true);
                   return true;
               }
               if (expr instanceof Slot && !partitionSlots.contains(expr)) {
                   containsUnEvaluableExpression.set(true);
                   return true;
               }
               // continue traverse children
               return false;
           });
           return containsUnEvaluableExpression.get();
       }
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] New expression extractor for partitions pruning. [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2172178542

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 172407 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 91e523aa1074b21dfe00fcc978f6af9167fc70fb, data reload: false
   
   query1	920	387	365	365
   query2	6430	2350	2216	2216
   query3	6634	209	209	209
   query4	19178	17307	17302	17302
   query5	3614	460	441	441
   query6	236	158	161	158
   query7	4613	297	289	289
   query8	337	289	279	279
   query9	8702	2375	2365	2365
   query10	571	297	273	273
   query11	10780	10205	10104	10104
   query12	131	83	85	83
   query13	1645	363	362	362
   query14	9945	6724	6071	6071
   query15	224	190	187	187
   query16	7210	299	277	277
   query17	1423	534	512	512
   query18	1857	270	265	265
   query19	195	155	158	155
   query20	91	85	82	82
   query21	209	132	127	127
   query22	4375	4027	4077	4027
   query23	33768	33614	33733	33614
   query24	11217	2944	2891	2891
   query25	617	363	400	363
   query26	1177	161	164	161
   query27	2474	336	333	333
   query28	7106	2126	2108	2108
   query29	894	661	608	608
   query30	254	151	152	151
   query31	968	759	780	759
   query32	91	52	54	52
   query33	746	282	277	277
   query34	1066	494	487	487
   query35	748	675	656	656
   query36	1134	951	982	951
   query37	172	72	77	72
   query38	2929	2847	2842	2842
   query39	894	843	839	839
   query40	206	135	130	130
   query41	58	52	55	52
   query42	108	104	106	104
   query43	585	543	532	532
   query44	1178	721	731	721
   query45	199	155	162	155
   query46	1072	728	695	695
   query47	1885	1799	1816	1799
   query48	349	291	289	289
   query49	849	384	394	384
   query50	756	396	392	392
   query51	6835	6610	6639	6610
   query52	104	88	88	88
   query53	350	287	289	287
   query54	865	445	433	433
   query55	76	71	74	71
   query56	299	252	255	252
   query57	1138	1036	1048	1036
   query58	245	245	228	228
   query59	3372	2957	2960	2957
   query60	282	268	272	268
   query61	90	93	88	88
   query62	614	425	449	425
   query63	311	293	283	283
   query64	8834	2345	1839	1839
   query65	3156	3120	3091	3091
   query66	824	335	344	335
   query67	15501	15033	14860	14860
   query68	5239	551	564	551
   query69	606	434	355	355
   query70	1158	1146	1079	1079
   query71	454	279	275	275
   query72	8542	5569	5410	5410
   query73	777	331	335	331
   query74	5983	5591	5554	5554
   query75	3668	2634	2670	2634
   query76	3086	957	943	943
   query77	614	291	292	291
   query78	10454	9815	9952	9815
   query79	5710	498	516	498
   query80	1981	464	458	458
   query81	597	212	227	212
   query82	1539	102	101	101
   query83	317	165	162	162
   query84	279	80	91	80
   query85	1554	275	263	263
   query86	546	312	291	291
   query87	3327	3108	3043	3043
   query88	5151	2332	2324	2324
   query89	484	376	379	376
   query90	1757	190	186	186
   query91	127	98	97	97
   query92	58	50	47	47
   query93	5605	509	508	508
   query94	1174	192	182	182
   query95	404	315	312	312
   query96	595	263	266	263
   query97	3233	3049	3021	3021
   query98	209	203	197	197
   query99	1313	856	831	831
   Total cold run time: 282257 ms
   Total hot run time: 172407 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173116080

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.8 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit b0a828dcb1cfd5b422c86356f5ee5b1c4d534a54, data reload: false
   
   query1	0.04	0.03	0.03
   query2	0.08	0.04	0.03
   query3	0.22	0.06	0.05
   query4	1.67	0.08	0.08
   query5	0.50	0.48	0.48
   query6	1.12	0.72	0.72
   query7	0.02	0.01	0.01
   query8	0.05	0.04	0.05
   query9	0.54	0.48	0.49
   query10	0.55	0.55	0.54
   query11	0.16	0.11	0.11
   query12	0.15	0.12	0.13
   query13	0.59	0.58	0.58
   query14	0.78	0.76	0.80
   query15	0.86	0.81	0.81
   query16	0.38	0.37	0.36
   query17	1.04	1.02	0.98
   query18	0.20	0.26	0.23
   query19	1.77	1.67	1.72
   query20	0.01	0.01	0.01
   query21	15.42	0.64	0.64
   query22	4.12	6.84	2.21
   query23	18.32	1.38	1.32
   query24	2.76	0.28	0.22
   query25	0.20	0.09	0.08
   query26	0.27	0.18	0.18
   query27	0.08	0.08	0.08
   query28	13.36	1.02	1.01
   query29	12.66	3.29	3.30
   query30	0.26	0.06	0.05
   query31	2.87	0.39	0.39
   query32	3.27	0.47	0.48
   query33	2.81	2.86	2.93
   query34	16.96	4.41	4.42
   query35	4.46	4.45	4.48
   query36	0.65	0.46	0.46
   query37	0.18	0.16	0.15
   query38	0.15	0.14	0.14
   query39	0.04	0.04	0.03
   query40	0.18	0.14	0.15
   query41	0.10	0.05	0.05
   query42	0.05	0.05	0.05
   query43	0.05	0.04	0.03
   Total cold run time: 109.95 s
   Total hot run time: 30.8 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]

Posted by "XuPengfei-1020 (via GitHub)" <gi...@apache.org>.
XuPengfei-1020 commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2175004591

   > could u add a case that can reproduce this problem?
   
   The case that can reporduce the issue at above, "Proposed changes" chapter (Added by @924060929 ).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] New expression extractor for partitions pruning. [doris]

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #36326:
URL: https://github.com/apache/doris/pull/36326#discussion_r1642139159


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruneExpressionExtractor.java:
##########
@@ -0,0 +1,204 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.rules.expression.rules.PartitionPruneExpressionExtractor.Context;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SubqueryExpr;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * PartitionPruneExpressionExtractor
+ *
+ * This rewriter only used to extract the expression that can be used in partition pruning from
+ *     the whole predicate expression.
+ * The theory of extractor is pretty simple:
+ * A:Sort the expression in two kinds:
+ *  1. evaluable-expression (let's mark it as E).
+ *      Expressions that can be evaluated in the partition pruning stage.
+ *      In the other word: not contains non-partition slots or deterministic expression.
+ *  2. un-evaluable-expression (let's mark it as UE).
+ *      Expressions that can NOT be evaluated in the partition pruning stage.
+ *      In the other word: contains non-partition slots or deterministic expression.
+ *
+ * B: Travel the predicate, only point on AND and OR operator, following the rule:
+ *    (E and UE) -> (E and TRUE) -> E
+ *    (UE and UE) -> TRUE
+ *    (E and E) -> (E and E)
+ *    (E or UE) -> TRUE
+ *    (UE or UE) -> TRUE
+ *    (E or E) -> (E or E)
+ *
+ * e.g.
+ *    (part = 1 and non_part = 'a') or (part = 2)
+ * -> (part = 1 and true) or (part = 2)
+ * -> (part = 1) or (part = 2)
+ *
+ * It's better that do some expression optimize(like fold, eliminate etc.) on predicate before this step.
+ */
+public class PartitionPruneExpressionExtractor extends DefaultExpressionRewriter<Context> {
+    private final ExpressionEvaluableDetector expressionEvaluableDetector;
+
+    private PartitionPruneExpressionExtractor(Set<Slot> interestedSlots, CascadesContext cascadesContext) {
+        this.expressionEvaluableDetector = new ExpressionEvaluableDetector(interestedSlots);
+    }
+
+    /**
+     * Extract partition prune expression from predicate
+     */
+    public static Expression extract(Expression predicate,
+                                     Set<Slot> partitionSlots,
+                                     CascadesContext cascadesContext) {
+        predicate = predicate.accept(FoldConstantRuleOnFE.VISITOR_INSTANCE, null);

Review Comment:
   new an ExpressionRewriteContext as context parameter:
   ```java
   predicate = predicate.accept(FoldConstantRuleOnFE.VISITOR_INSTANCE, new ExpressionRewriteContext(cascadesContext));
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173085290

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 39594 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit b0a828dcb1cfd5b422c86356f5ee5b1c4d534a54, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17631	4332	4237	4237
   q2	2029	190	198	190
   q3	10427	1197	1107	1107
   q4	10187	804	734	734
   q5	7449	2635	2602	2602
   q6	221	134	135	134
   q7	960	610	595	595
   q8	9214	2032	2072	2032
   q9	8831	6456	6442	6442
   q10	8905	3712	3766	3712
   q11	452	231	240	231
   q12	476	231	224	224
   q13	17777	2981	2967	2967
   q14	265	228	224	224
   q15	505	475	475	475
   q16	516	371	382	371
   q17	949	659	677	659
   q18	8058	7486	7378	7378
   q19	2204	1488	1536	1488
   q20	652	298	324	298
   q21	4914	3162	3788	3162
   q22	386	332	339	332
   Total cold run time: 113008 ms
   Total hot run time: 39594 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4335	4179	4259	4179
   q2	373	273	274	273
   q3	2945	2705	2707	2705
   q4	2096	1732	1684	1684
   q5	5441	5618	5488	5488
   q6	227	127	131	127
   q7	2180	1823	1813	1813
   q8	3276	3419	3388	3388
   q9	8685	8687	8735	8687
   q10	4127	3860	3879	3860
   q11	579	485	485	485
   q12	774	578	599	578
   q13	16150	3227	3184	3184
   q14	298	274	291	274
   q15	520	468	501	468
   q16	504	440	446	440
   q17	1784	1515	1491	1491
   q18	8080	7829	7825	7825
   q19	2831	1707	1637	1637
   q20	2138	1886	1869	1869
   q21	5034	4867	4890	4867
   q22	638	593	557	557
   Total cold run time: 73015 ms
   Total hot run time: 55879 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173431850

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.62 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 0c072d3adaecd053725e993dce322a63de6a8e9d, data reload: false
   
   query1	0.04	0.03	0.03
   query2	0.08	0.04	0.04
   query3	0.23	0.04	0.04
   query4	1.69	0.07	0.07
   query5	0.51	0.52	0.49
   query6	1.14	0.74	0.72
   query7	0.02	0.02	0.01
   query8	0.05	0.04	0.04
   query9	0.55	0.49	0.48
   query10	0.54	0.54	0.54
   query11	0.15	0.11	0.12
   query12	0.15	0.12	0.12
   query13	0.59	0.59	0.60
   query14	0.79	0.79	0.77
   query15	0.84	0.81	0.81
   query16	0.36	0.37	0.36
   query17	0.96	1.06	1.05
   query18	0.22	0.24	0.25
   query19	1.90	1.70	1.72
   query20	0.02	0.01	0.01
   query21	15.41	0.67	0.66
   query22	4.28	6.96	2.00
   query23	18.26	1.25	1.24
   query24	2.18	0.22	0.21
   query25	0.16	0.08	0.09
   query26	0.26	0.18	0.18
   query27	0.09	0.08	0.09
   query28	13.17	1.02	1.00
   query29	12.61	3.22	3.19
   query30	0.26	0.06	0.06
   query31	2.86	0.38	0.39
   query32	3.29	0.49	0.47
   query33	2.91	2.87	2.89
   query34	17.18	4.46	4.42
   query35	4.48	4.53	4.59
   query36	0.66	0.48	0.46
   query37	0.18	0.15	0.15
   query38	0.15	0.14	0.15
   query39	0.04	0.03	0.04
   query40	0.17	0.13	0.13
   query41	0.10	0.06	0.04
   query42	0.06	0.05	0.04
   query43	0.05	0.04	0.04
   Total cold run time: 109.64 s
   Total hot run time: 30.62 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173337884

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]

Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2174875101

   could u add a case that can reproduce this problem?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [refactor](Nereids) New expression extractor for partitions pruning [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36326:
URL: https://github.com/apache/doris/pull/36326#issuecomment-2173111157

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org