You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "seawinde (via GitHub)" <gi...@apache.org> on 2024/01/26 09:53:45 UTC

[PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

seawinde opened a new pull request, #30440:
URL: https://github.com/apache/doris/pull/30440

   ## Proposed changes
   
   
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1914071675

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 31.51 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit d90e1c0495d37fbb7a923da38d25246b3963c7b1, data reload: false
   
   query1	0.04	0.03	0.03
   query2	0.06	0.02	0.03
   query3	0.23	0.06	0.07
   query4	1.65	0.11	0.10
   query5	0.52	0.51	0.52
   query6	1.19	0.63	0.64
   query7	0.02	0.02	0.01
   query8	0.04	0.03	0.03
   query9	0.53	0.51	0.49
   query10	0.53	0.55	0.55
   query11	0.12	0.08	0.08
   query12	0.10	0.09	0.10
   query13	0.61	0.60	0.60
   query14	0.79	0.83	0.80
   query15	0.79	0.78	0.77
   query16	0.38	0.38	0.38
   query17	1.03	0.98	0.96
   query18	0.21	0.23	0.27
   query19	1.86	1.76	1.78
   query20	0.01	0.01	0.01
   query21	15.42	0.59	0.58
   query22	2.61	2.70	2.46
   query23	17.25	0.84	0.86
   query24	2.50	1.12	0.90
   query25	0.29	0.29	0.09
   query26	0.45	0.14	0.15
   query27	0.05	0.05	0.04
   query28	11.31	0.87	0.85
   query29	12.57	3.17	3.25
   query30	0.64	0.56	0.54
   query31	2.79	0.34	0.35
   query32	3.34	0.48	0.47
   query33	3.22	3.25	3.20
   query34	15.72	4.24	4.33
   query35	4.25	4.24	4.28
   query36	1.10	1.06	1.07
   query37	0.06	0.05	0.05
   query38	0.04	0.03	0.02
   query39	0.03	0.01	0.02
   query40	0.16	0.12	0.13
   query41	0.07	0.01	0.02
   query42	0.02	0.01	0.02
   query43	0.03	0.02	0.02
   Total cold run time: 104.63 s
   Total hot run time: 31.51 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1913526189

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 37849 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 019c33c82e8e7b7931328b89e06fbb8bdd8559d7, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17883	5220	5015	5015
   q2	2877	144	138	138
   q3	11537	939	956	939
   q4	4957	828	757	757
   q5	7902	2871	2895	2871
   q6	192	125	124	124
   q7	1175	756	759	756
   q8	9488	2032	2061	2032
   q9	7200	6354	6346	6346
   q10	8106	2430	2415	2415
   q11	415	202	210	202
   q12	770	269	283	269
   q13	18024	3310	3326	3310
   q14	270	254	253	253
   q15	542	492	483	483
   q16	460	399	393	393
   q17	946	543	538	538
   q18	7014	5964	6196	5964
   q19	1581	1445	1307	1307
   q20	602	328	333	328
   q21	6797	3127	3117	3117
   q22	824	292	292	292
   Total cold run time: 109562 ms
   Total hot run time: 37849 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4986	5180	4985	4985
   q2	343	255	253	253
   q3	3038	2691	2712	2691
   q4	1861	1627	1556	1556
   q5	5460	5252	5196	5196
   q6	207	116	119	116
   q7	2181	1735	1845	1735
   q8	3122	3212	3246	3212
   q9	8686	8521	11385	8521
   q10	5421	3475	3484	3475
   q11	549	480	459	459
   q12	761	593	610	593
   q13	16963	3135	3052	3052
   q14	275	250	266	250
   q15	531	491	484	484
   q16	523	483	468	468
   q17	2100	1627	1689	1627
   q18	9007	15413	8954	8954
   q19	26879	1594	1511	1511
   q20	4640	1903	1905	1903
   q21	16061	4820	4824	4820
   q22	1537	461	488	461
   Total cold run time: 115131 ms
   Total hot run time: 56322 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1914023689

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1913893098

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on code in PR #30440:
URL: https://github.com/apache/doris/pull/30440#discussion_r1469041729


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewAggregateRule.java:
##########
@@ -438,9 +439,110 @@ private boolean isAggregateFunctionEquivalent(Function queryFunction, Function v
      * actualFunction is bitmap_union(to_bitmap(case when a = 5 then 1 else 2 end))
      * after extracting, the return argument is: case when a = 5 then 1 else 2 end
      */
-    private List<Expression> extractArguments(Expression functionWithAny, Function actualFunction) {
+    private static List<Expression> extractArguments(Expression functionWithAny, Function actualFunction) {
         Set<Object> exprSetToRemove = functionWithAny.collectToSet(expr -> !(expr instanceof Any));
         return actualFunction.collectFirst(expr ->
                 exprSetToRemove.stream().noneMatch(exprToRemove -> exprToRemove.equals(expr)));
     }
+
+    /**
+     * Aggregate expression rewriter which is responsible for rewriting group by and
+     * aggregate function expression
+     */
+    protected static class AggregateExpressionRewriter
+            extends DefaultExpressionRewriter<AggregateExpressionRewriteContext> {
+
+        @Override
+        public Expression visitAggregateFunction(AggregateFunction aggregateFunction,
+                AggregateExpressionRewriteContext rewriteContext) {
+            if (!rewriteContext.isValid()) {
+                return null;
+            }
+            Expression queryFunctionShuttled = ExpressionUtils.shuttleExpressionWithLineage(
+                    aggregateFunction,
+                    rewriteContext.getQueryTopPlan());
+            Function rollupAggregateFunction = rollup(aggregateFunction, queryFunctionShuttled,
+                    rewriteContext.getMvExprToMvScanExprQueryBased());
+            if (rollupAggregateFunction == null) {
+                rewriteContext.setValid(false);
+                return null;
+            }
+            return rollupAggregateFunction;
+        }
+
+        @Override
+        public Expression visitSlot(Slot slot, AggregateExpressionRewriteContext rewriteContext) {
+            if (!rewriteContext.isValid()) {
+                return null;
+            }
+            if (rewriteContext.getMvExprToMvScanExprQueryBased().containsKey(slot)) {
+                return rewriteContext.getMvExprToMvScanExprQueryBased().get(slot);
+            }
+            rewriteContext.setValid(false);
+            return null;
+        }

Review Comment:
   ```suggestion
               return originExpr;
           }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1913893072

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1914065730

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 172972 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit d90e1c0495d37fbb7a923da38d25246b3963c7b1, data reload: false
   
   query1	950	334	323	323
   query2	6585	2062	1844	1844
   query3	6699	198	196	196
   query4	33409	21979	22111	21979
   query5	4448	419	348	348
   query6	244	151	148	148
   query7	4605	261	265	261
   query8	244	168	168	168
   query9	8571	2264	2241	2241
   query10	409	202	207	202
   query11	17805	15529	15288	15288
   query12	114	66	67	66
   query13	1675	365	373	365
   query14	9123	6543	6522	6522
   query15	210	179	177	177
   query16	5701	256	255	255
   query17	946	475	468	468
   query18	1804	256	250	250
   query19	181	134	135	134
   query20	74	63	63	63
   query21	205	135	129	129
   query22	4866	4712	4742	4712
   query23	31127	30444	30433	30433
   query24	11619	2763	2805	2763
   query25	590	334	300	300
   query26	1818	136	135	135
   query27	3201	283	280	280
   query28	7555	1838	1813	1813
   query29	1867	611	590	590
   query30	282	138	136	136
   query31	890	715	709	709
   query32	77	52	48	48
   query33	702	215	201	201
   query34	1160	451	460	451
   query35	923	754	739	739
   query36	1296	1224	1214	1214
   query37	91	54	57	54
   query38	3252	3186	3168	3168
   query39	1290	1252	1243	1243
   query40	339	87	84	84
   query41	38	38	34	34
   query42	91	80	82	80
   query43	528	473	500	473
   query44	1133	686	702	686
   query45	193	175	170	170
   query46	1070	654	653	653
   query47	1604	1498	1518	1498
   query48	386	312	320	312
   query49	1238	282	277	277
   query50	687	311	313	311
   query51	5348	5188	5209	5188
   query52	97	83	78	78
   query53	316	257	262	257
   query54	253	184	184	184
   query55	83	69	72	69
   query56	185	156	157	156
   query57	963	887	904	887
   query58	190	159	154	154
   query59	2599	2587	2380	2380
   query60	198	177	177	177
   query61	85	86	80	80
   query62	605	355	363	355
   query63	286	274	255	255
   query64	5970	1743	1721	1721
   query65	3271	3230	3223	3223
   query66	1240	323	309	309
   query67	14528	14195	14089	14089
   query68	4041	496	476	476
   query69	420	297	302	297
   query70	1506	1482	1495	1482
   query71	253	230	211	211
   query72	3799	2846	2838	2838
   query73	688	321	314	314
   query74	6764	6209	6303	6209
   query75	2858	2321	2308	2308
   query76	2587	1069	1046	1046
   query77	368	227	228	227
   query78	9124	8770	8589	8589
   query79	2009	490	488	488
   query80	1373	320	307	307
   query81	518	197	201	197
   query82	678	78	77	77
   query83	229	117	110	110
   query84	276	71	78	71
   query85	2119	335	326	326
   query86	521	404	401	401
   query87	3434	3276	3303	3276
   query88	3913	2184	2178	2178
   query89	415	360	365	360
   query90	1875	188	188	188
   query91	152	114	120	114
   query92	55	42	43	42
   query93	2112	420	436	420
   query94	1355	158	160	158
   query95	497	473	437	437
   query96	619	314	315	314
   query97	4275	4118	4127	4118
   query98	202	198	184	184
   query99	1074	671	653	653
   Total cold run time: 280241 ms
   Total hot run time: 172972 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1911776287

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on code in PR #30440:
URL: https://github.com/apache/doris/pull/30440#discussion_r1469133541


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewAggregateRule.java:
##########
@@ -438,9 +439,110 @@ private boolean isAggregateFunctionEquivalent(Function queryFunction, Function v
      * actualFunction is bitmap_union(to_bitmap(case when a = 5 then 1 else 2 end))
      * after extracting, the return argument is: case when a = 5 then 1 else 2 end
      */
-    private List<Expression> extractArguments(Expression functionWithAny, Function actualFunction) {
+    private static List<Expression> extractArguments(Expression functionWithAny, Function actualFunction) {
         Set<Object> exprSetToRemove = functionWithAny.collectToSet(expr -> !(expr instanceof Any));
         return actualFunction.collectFirst(expr ->
                 exprSetToRemove.stream().noneMatch(exprToRemove -> exprToRemove.equals(expr)));
     }
+
+    /**
+     * Aggregate expression rewriter which is responsible for rewriting group by and
+     * aggregate function expression
+     */
+    protected static class AggregateExpressionRewriter
+            extends DefaultExpressionRewriter<AggregateExpressionRewriteContext> {
+
+        @Override
+        public Expression visitAggregateFunction(AggregateFunction aggregateFunction,
+                AggregateExpressionRewriteContext rewriteContext) {
+            if (!rewriteContext.isValid()) {
+                return null;
+            }
+            Expression queryFunctionShuttled = ExpressionUtils.shuttleExpressionWithLineage(
+                    aggregateFunction,
+                    rewriteContext.getQueryTopPlan());
+            Function rollupAggregateFunction = rollup(aggregateFunction, queryFunctionShuttled,
+                    rewriteContext.getMvExprToMvScanExprQueryBased());
+            if (rollupAggregateFunction == null) {
+                rewriteContext.setValid(false);
+                return null;
+            }
+            return rollupAggregateFunction;
+        }
+
+        @Override
+        public Expression visitSlot(Slot slot, AggregateExpressionRewriteContext rewriteContext) {
+            if (!rewriteContext.isValid()) {
+                return null;
+            }
+            if (rewriteContext.getMvExprToMvScanExprQueryBased().containsKey(slot)) {
+                return rewriteContext.getMvExprToMvScanExprQueryBased().get(slot);
+            }
+            rewriteContext.setValid(false);
+            return null;
+        }

Review Comment:
   Have fixed it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1913529800

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 172622 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 019c33c82e8e7b7931328b89e06fbb8bdd8559d7, data reload: false
   
   query1	926	335	330	330
   query2	6545	1897	2048	1897
   query3	6709	200	199	199
   query4	32579	22154	22121	22121
   query5	4447	410	342	342
   query6	254	159	156	156
   query7	4600	263	257	257
   query8	240	180	170	170
   query9	8582	2269	2244	2244
   query10	422	214	199	199
   query11	17874	15516	15309	15309
   query12	110	66	64	64
   query13	1692	379	378	378
   query14	9202	6378	6479	6378
   query15	217	185	182	182
   query16	5711	259	243	243
   query17	951	479	463	463
   query18	1795	248	253	248
   query19	169	134	130	130
   query20	75	64	68	64
   query21	203	134	135	134
   query22	4678	4881	4516	4516
   query23	31244	30397	30309	30309
   query24	12421	2752	2756	2752
   query25	599	313	307	307
   query26	1973	144	137	137
   query27	3181	284	282	282
   query28	7741	1820	1810	1810
   query29	2248	598	608	598
   query30	290	135	134	134
   query31	892	689	721	689
   query32	75	47	50	47
   query33	712	213	210	210
   query34	1163	450	471	450
   query35	849	756	775	756
   query36	1319	1139	1120	1120
   query37	98	59	60	59
   query38	3243	3161	3120	3120
   query39	1327	1252	1256	1252
   query40	342	89	82	82
   query41	38	35	34	34
   query42	89	79	83	79
   query43	546	501	505	501
   query44	1101	679	686	679
   query45	191	175	172	172
   query46	1033	659	653	653
   query47	1621	1446	1547	1446
   query48	376	326	300	300
   query49	1225	283	286	283
   query50	668	315	316	315
   query51	5259	5124	5180	5124
   query52	85	78	80	78
   query53	323	257	265	257
   query54	238	186	186	186
   query55	76	77	74	74
   query56	174	155	167	155
   query57	1039	914	903	903
   query58	185	150	152	150
   query59	2800	2733	2525	2525
   query60	202	176	179	176
   query61	81	84	81	81
   query62	651	365	367	365
   query63	279	268	268	268
   query64	5985	1752	1743	1743
   query65	3300	3225	3224	3224
   query66	1457	316	313	313
   query67	14095	14383	14221	14221
   query68	3741	495	498	495
   query69	417	305	295	295
   query70	1539	1456	1493	1456
   query71	292	208	240	208
   query72	3671	2836	2847	2836
   query73	699	316	322	316
   query74	6666	6192	6289	6192
   query75	2843	2314	2310	2310
   query76	2610	1015	1026	1015
   query77	427	235	228	228
   query78	9006	8705	8501	8501
   query79	2117	482	500	482
   query80	1383	322	329	322
   query81	522	202	200	200
   query82	680	84	78	78
   query83	231	113	123	113
   query84	279	79	73	73
   query85	2201	352	323	323
   query86	500	399	380	380
   query87	3374	3256	3284	3256
   query88	3872	2150	2138	2138
   query89	421	354	359	354
   query90	2052	186	181	181
   query91	151	116	119	116
   query92	56	43	44	43
   query93	2125	440	409	409
   query94	1471	162	159	159
   query95	496	455	463	455
   query96	626	304	316	304
   query97	4243	4090	4101	4090
   query98	206	204	192	192
   query99	1049	659	688	659
   Total cold run time: 280781 ms
   Total hot run time: 172622 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1914054541

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 37540 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit d90e1c0495d37fbb7a923da38d25246b3963c7b1, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17647	5306	4936	4936
   q2	2044	143	131	131
   q3	10696	902	929	902
   q4	4699	679	724	679
   q5	7664	2878	2822	2822
   q6	185	117	118	117
   q7	1123	736	711	711
   q8	9339	1991	2024	1991
   q9	7262	6295	6283	6283
   q10	8101	2409	2428	2409
   q11	403	207	202	202
   q12	765	284	271	271
   q13	18004	3273	3305	3273
   q14	278	257	242	242
   q15	524	488	483	483
   q16	452	400	401	400
   q17	932	572	564	564
   q18	7150	6065	6038	6038
   q19	1576	1395	1389	1389
   q20	577	328	352	328
   q21	6951	3083	3113	3083
   q22	803	286	290	286
   Total cold run time: 107175 ms
   Total hot run time: 37540 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	5175	5000	5022	5000
   q2	352	248	238	238
   q3	2993	2662	2690	2662
   q4	1833	1512	1576	1512
   q5	5271	5163	5243	5163
   q6	202	113	116	113
   q7	2112	1761	1737	1737
   q8	3160	3159	3185	3159
   q9	8608	8635	10620	8635
   q10	5480	3436	3468	3436
   q11	546	466	465	465
   q12	765	642	583	583
   q13	16915	3112	3065	3065
   q14	283	250	265	250
   q15	531	483	493	483
   q16	519	475	464	464
   q17	2019	1674	1570	1570
   q18	8868	10215	8562	8562
   q19	24259	1604	1539	1539
   q20	4627	1897	1878	1878
   q21	15820	4840	4792	4792
   q22	1113	472	502	472
   Total cold run time: 111451 ms
   Total hot run time: 55778 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1914030662

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #30440:
URL: https://github.com/apache/doris/pull/30440


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1913531205

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.96 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 019c33c82e8e7b7931328b89e06fbb8bdd8559d7, data reload: false
   
   query1	0.03	0.04	0.03
   query2	0.06	0.02	0.03
   query3	0.23	0.06	0.06
   query4	1.66	0.11	0.10
   query5	0.52	0.52	0.50
   query6	1.19	0.62	0.61
   query7	0.02	0.02	0.01
   query8	0.04	0.02	0.03
   query9	0.55	0.49	0.50
   query10	0.57	0.55	0.56
   query11	0.12	0.09	0.08
   query12	0.11	0.09	0.10
   query13	0.59	0.60	0.60
   query14	0.79	0.79	0.81
   query15	0.79	0.78	0.78
   query16	0.38	0.39	0.37
   query17	1.00	1.03	1.03
   query18	0.25	0.24	0.27
   query19	1.89	1.79	1.79
   query20	0.01	0.01	0.01
   query21	15.41	0.55	0.57
   query22	2.66	2.41	2.09
   query23	17.29	0.91	0.72
   query24	2.42	1.65	0.45
   query25	0.32	0.25	0.23
   query26	0.49	0.13	0.14
   query27	0.06	0.04	0.04
   query28	11.40	0.84	0.86
   query29	12.53	3.20	3.21
   query30	0.67	0.55	0.60
   query31	2.79	0.34	0.36
   query32	3.36	0.48	0.48
   query33	3.20	3.22	3.24
   query34	16.00	4.29	4.32
   query35	4.44	4.36	4.37
   query36	1.09	1.05	1.06
   query37	0.07	0.05	0.05
   query38	0.04	0.03	0.02
   query39	0.02	0.02	0.02
   query40	0.17	0.13	0.12
   query41	0.07	0.01	0.01
   query42	0.02	0.01	0.01
   query43	0.03	0.02	0.02
   Total cold run time: 105.35 s
   Total hot run time: 30.96 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1913532105

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit 019c33c82e8e7b7931328b89e06fbb8bdd8559d7 with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1914075115

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit d90e1c0495d37fbb7a923da38d25246b3963c7b1 with default session variables
   Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
   Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
   Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
   Insert into select:       13.8 seconds inserted 10000000 Rows, about 724K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]

Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1913519851

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org