You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "starocean999 (via GitHub)" <gi...@apache.org> on 2024/06/05 04:18:45 UTC

[PR] [fix](nereids)disable unique key compute for mv [doris]

starocean999 opened a new pull request, #35898:
URL: https://github.com/apache/doris/pull/35898

   ## Proposed changes
   
   computing unique doesn't work for mv, because mv's key may be different from base table and the key can be any expression which is difficult to deduce if it's unique. for example
   base table:
   ```
   CREATE TABLE IF NOT EXISTS base(
       siteid INT(11) NOT NULL,
       citycode SMALLINT(6) NOT NULL,
       username VARCHAR(32) NOT NULL,
       pv BIGINT(20) SUM NOT NULL DEFAULT '0'
   )
   AGGREGATE KEY (siteid,citycode,username)
   DISTRIBUTED BY HASH(siteid) BUCKETS 5 properties("replication_num" = "1");
   ```
   case1:
   create mv1:
   `create materialized view mv1 as select siteid, sum(pv) from base group by siteid;`
   the base table siteid + citycode + username is unique but the mv1's agg key siteid is not unique
   
   case2:
   create mv2:
   `create materialized view mv2 as select citycode * citycode, siteid, username from base;`
   the mv2's agg key citycode * citycode is not unique
   
   for simplicity, we disable unique compute for mv
   
   <!--Describe your changes.-->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids)disable unique key compute for mv [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2151328964

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids)disable unique key compute for mv [doris]

Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on code in PR #35898:
URL: https://github.com/apache/doris/pull/35898#discussion_r1628714500


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java:
##########
@@ -128,20 +127,6 @@ public String qualifiedName() {
     @Override
     public void computeUnique(DataTrait.Builder builder) {
         Set<Slot> outputSet = Utils.fastToImmutableSet(getOutputSet());
-        if (table instanceof OlapTable && ((OlapTable) table).getKeysType().isAggregationFamily()) {
-            ImmutableSet.Builder<Slot> uniqSlots = ImmutableSet.builderWithExpectedSize(outputSet.size());
-            for (Slot slot : outputSet) {
-                if (!(slot instanceof SlotReference)) {
-                    continue;
-                }
-                SlotReference slotRef = (SlotReference) slot;
-                if (slotRef.getColumn().isPresent() && slotRef.getColumn().get().isKey()) {
-                    uniqSlots.add(slot);
-                }
-            }
-            builder.addUniqueSlot(uniqSlots.build());
-        }
-

Review Comment:
   Add a condition to check if it is a mv rather than remove those codes. They are useful if it's a trivial olapscan



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids)disable unique key compute for mv [doris]

Posted by "starocean999 (via GitHub)" <gi...@apache.org>.
starocean999 merged PR #35898:
URL: https://github.com/apache/doris/pull/35898


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids)disable unique key compute for mv [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2172849013

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 172208 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 94f4de5c91b98645a878ff189333f169da09df94, data reload: false
   
   query1	922	389	373	373
   query2	6473	2424	2309	2309
   query3	6648	209	212	209
   query4	19461	17112	17271	17112
   query5	4148	482	467	467
   query6	269	167	172	167
   query7	4595	304	294	294
   query8	314	293	283	283
   query9	8539	2421	2388	2388
   query10	610	305	290	290
   query11	10615	10043	9961	9961
   query12	140	91	84	84
   query13	1656	390	367	367
   query14	10128	6988	7043	6988
   query15	227	192	194	192
   query16	7740	259	260	259
   query17	1843	539	568	539
   query18	1942	283	273	273
   query19	188	153	156	153
   query20	93	88	88	88
   query21	211	134	130	130
   query22	4279	4030	3994	3994
   query23	33524	32972	33004	32972
   query24	12069	2914	2837	2837
   query25	652	353	369	353
   query26	1810	161	154	154
   query27	2929	326	326	326
   query28	7195	2064	2031	2031
   query29	1100	634	623	623
   query30	284	153	155	153
   query31	936	737	735	735
   query32	86	55	56	55
   query33	772	305	295	295
   query34	928	511	476	476
   query35	740	624	605	605
   query36	1063	937	956	937
   query37	292	70	72	70
   query38	2867	2730	2700	2700
   query39	863	793	770	770
   query40	289	127	127	127
   query41	58	56	56	56
   query42	126	98	107	98
   query43	607	536	542	536
   query44	1218	729	729	729
   query45	198	164	165	164
   query46	1076	743	726	726
   query47	1838	1789	1755	1755
   query48	383	311	301	301
   query49	1214	404	402	402
   query50	769	401	397	397
   query51	6913	6757	6583	6583
   query52	102	95	92	92
   query53	364	288	291	288
   query54	989	463	445	445
   query55	77	76	75	75
   query56	286	248	251	248
   query57	1170	1048	1069	1048
   query58	254	282	240	240
   query59	3397	3260	3058	3058
   query60	293	278	284	278
   query61	97	90	92	90
   query62	657	445	444	444
   query63	319	296	290	290
   query64	9928	2300	1740	1740
   query65	3234	3087	3129	3087
   query66	1362	343	340	340
   query67	15500	14938	14771	14771
   query68	4576	543	544	543
   query69	463	318	309	309
   query70	1120	1144	1116	1116
   query71	399	282	269	269
   query72	7196	5350	5545	5350
   query73	751	322	327	322
   query74	5881	5469	5512	5469
   query75	3365	2732	2689	2689
   query76	2761	929	929	929
   query77	444	315	294	294
   query78	10197	9847	9821	9821
   query79	2281	525	525	525
   query80	830	473	461	461
   query81	567	233	225	225
   query82	768	106	101	101
   query83	284	179	182	179
   query84	241	93	87	87
   query85	1264	284	268	268
   query86	458	320	338	320
   query87	3278	3079	3121	3079
   query88	4183	2406	2377	2377
   query89	479	390	400	390
   query90	1782	264	197	197
   query91	133	98	99	98
   query92	67	50	47	47
   query93	1781	515	502	502
   query94	1259	190	191	190
   query95	398	311	320	311
   query96	599	271	271	271
   query97	3211	3047	3028	3028
   query98	215	202	189	189
   query99	1149	836	832	832
   Total cold run time: 274249 ms
   Total hot run time: 172208 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids)disable unique key compute for mv [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2172859510

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.5 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 94f4de5c91b98645a878ff189333f169da09df94, data reload: false
   
   query1	0.04	0.03	0.03
   query2	0.08	0.04	0.04
   query3	0.23	0.05	0.05
   query4	1.73	0.08	0.10
   query5	0.49	0.48	0.48
   query6	1.14	0.73	0.73
   query7	0.02	0.01	0.02
   query8	0.05	0.05	0.05
   query9	0.53	0.49	0.49
   query10	0.56	0.55	0.53
   query11	0.16	0.11	0.12
   query12	0.14	0.11	0.12
   query13	0.60	0.59	0.60
   query14	0.79	0.77	0.77
   query15	0.86	0.81	0.80
   query16	0.38	0.35	0.35
   query17	0.99	0.97	0.97
   query18	0.24	0.25	0.27
   query19	1.82	1.70	1.68
   query20	0.01	0.01	0.01
   query21	15.41	0.64	0.64
   query22	4.37	7.01	1.94
   query23	18.28	1.42	1.28
   query24	2.09	0.22	0.21
   query25	0.15	0.08	0.08
   query26	0.26	0.18	0.17
   query27	0.09	0.08	0.07
   query28	13.26	1.03	0.99
   query29	12.63	3.38	3.32
   query30	0.26	0.06	0.06
   query31	2.86	0.39	0.39
   query32	3.28	0.48	0.47
   query33	2.90	2.92	2.87
   query34	17.07	4.39	4.38
   query35	4.51	4.56	4.44
   query36	0.65	0.46	0.47
   query37	0.18	0.16	0.16
   query38	0.16	0.15	0.15
   query39	0.04	0.03	0.04
   query40	0.17	0.16	0.14
   query41	0.10	0.04	0.04
   query42	0.05	0.04	0.05
   query43	0.04	0.04	0.04
   Total cold run time: 109.67 s
   Total hot run time: 30.5 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids)disable unique key compute for mv [doris]

Posted by "starocean999 (via GitHub)" <gi...@apache.org>.
starocean999 commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2148823650

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids)disable unique key compute for mv [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2148885765

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 169554 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 6c1ea4dcd61a2edb41cb8b87b7158ec4ce59929a, data reload: false
   
   query1	942	378	368	368
   query2	6460	2416	2299	2299
   query3	6651	208	209	208
   query4	20038	17184	17535	17184
   query5	4164	453	453	453
   query6	260	164	164	164
   query7	4593	306	283	283
   query8	315	279	278	278
   query9	8550	2419	2392	2392
   query10	463	316	286	286
   query11	10545	10072	9991	9991
   query12	137	87	85	85
   query13	1639	356	364	356
   query14	9211	7597	6790	6790
   query15	229	187	184	184
   query16	7823	259	263	259
   query17	1553	521	514	514
   query18	1945	284	281	281
   query19	193	151	148	148
   query20	91	87	91	87
   query21	205	136	131	131
   query22	4430	4212	3958	3958
   query23	33744	33060	33192	33060
   query24	12176	2810	2809	2809
   query25	661	352	358	352
   query26	1829	155	161	155
   query27	3066	317	321	317
   query28	7702	2048	2043	2043
   query29	1106	613	603	603
   query30	285	171	149	149
   query31	953	735	742	735
   query32	85	53	54	53
   query33	747	274	267	267
   query34	1014	482	475	475
   query35	753	615	629	615
   query36	1117	939	892	892
   query37	183	69	69	69
   query38	2909	2771	2756	2756
   query39	861	793	787	787
   query40	282	122	124	122
   query41	49	50	51	50
   query42	115	95	100	95
   query43	600	537	542	537
   query44	1232	727	752	727
   query45	207	173	165	165
   query46	1073	732	744	732
   query47	1866	1778	1791	1778
   query48	386	287	300	287
   query49	1180	407	443	407
   query50	779	391	394	391
   query51	6715	6681	6664	6664
   query52	101	104	87	87
   query53	352	291	289	289
   query54	984	439	433	433
   query55	76	75	73	73
   query56	271	250	253	250
   query57	1154	1035	1091	1035
   query58	252	265	239	239
   query59	3380	3224	3269	3224
   query60	294	264	269	264
   query61	91	86	118	86
   query62	647	448	456	448
   query63	317	296	284	284
   query64	9839	2231	1712	1712
   query65	3149	3094	3084	3084
   query66	1371	340	326	326
   query67	15442	14851	14895	14851
   query68	4539	544	533	533
   query69	465	309	308	308
   query70	1181	1126	1156	1126
   query71	394	281	289	281
   query72	7181	5886	2904	2904
   query73	742	329	321	321
   query74	5951	5587	5435	5435
   query75	3378	2649	2699	2649
   query76	2439	976	890	890
   query77	462	306	298	298
   query78	10447	9843	9651	9651
   query79	2941	512	509	509
   query80	1961	492	468	468
   query81	585	224	223	223
   query82	717	103	106	103
   query83	342	180	177	177
   query84	276	89	92	89
   query85	1889	330	325	325
   query86	482	323	311	311
   query87	3273	3069	3120	3069
   query88	3950	2358	2368	2358
   query89	499	397	386	386
   query90	1845	194	204	194
   query91	137	108	109	108
   query92	70	52	53	52
   query93	4447	521	501	501
   query94	1290	196	200	196
   query95	418	322	326	322
   query96	604	266	263	263
   query97	3227	3029	2993	2993
   query98	239	222	231	222
   query99	1375	813	819	813
   Total cold run time: 279578 ms
   Total hot run time: 169554 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids)disable unique key compute for mv [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2148892309

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.56 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 6c1ea4dcd61a2edb41cb8b87b7158ec4ce59929a, data reload: false
   
   query1	0.04	0.04	0.04
   query2	0.08	0.04	0.04
   query3	0.24	0.04	0.05
   query4	1.68	0.09	0.09
   query5	0.50	0.49	0.49
   query6	1.12	0.72	0.73
   query7	0.03	0.01	0.02
   query8	0.06	0.04	0.04
   query9	0.55	0.47	0.50
   query10	0.55	0.55	0.54
   query11	0.16	0.11	0.11
   query12	0.14	0.11	0.12
   query13	0.60	0.59	0.60
   query14	0.76	0.78	0.77
   query15	0.84	0.81	0.80
   query16	0.36	0.36	0.38
   query17	0.99	0.96	1.03
   query18	0.22	0.26	0.25
   query19	1.84	1.81	1.80
   query20	0.02	0.01	0.02
   query21	15.73	0.65	0.66
   query22	4.35	7.42	2.04
   query23	18.22	1.34	1.20
   query24	2.06	0.19	0.20
   query25	0.13	0.07	0.08
   query26	0.28	0.18	0.18
   query27	0.07	0.07	0.08
   query28	13.56	1.02	0.99
   query29	13.49	3.25	3.26
   query30	0.24	0.06	0.06
   query31	2.87	0.39	0.39
   query32	3.25	0.48	0.46
   query33	2.85	2.90	2.98
   query34	17.27	4.40	4.37
   query35	4.50	4.48	4.48
   query36	0.65	0.48	0.45
   query37	0.17	0.14	0.14
   query38	0.15	0.14	0.14
   query39	0.04	0.03	0.04
   query40	0.17	0.15	0.15
   query41	0.09	0.05	0.04
   query42	0.06	0.05	0.04
   query43	0.04	0.04	0.03
   Total cold run time: 111.02 s
   Total hot run time: 30.56 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids)disable unique key compute for mv [doris]

Posted by "starocean999 (via GitHub)" <gi...@apache.org>.
starocean999 commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2172593878

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids)disable unique key compute for mv [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2148823602

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
   See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids)disable unique key compute for mv [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2148873802

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 41019 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 6c1ea4dcd61a2edb41cb8b87b7158ec4ce59929a, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17735	4399	4295	4295
   q2	2034	189	195	189
   q3	10619	1234	1182	1182
   q4	10623	933	785	785
   q5	7528	2766	2770	2766
   q6	227	134	136	134
   q7	965	610	610	610
   q8	9257	2121	2091	2091
   q9	9656	6649	6633	6633
   q10	9726	3815	3723	3723
   q11	453	246	245	245
   q12	473	232	225	225
   q13	19032	2917	2997	2917
   q14	262	205	226	205
   q15	511	483	467	467
   q16	525	374	383	374
   q17	1000	722	718	718
   q18	8060	7424	7528	7424
   q19	3254	1495	1480	1480
   q20	661	314	309	309
   q21	4902	3939	3911	3911
   q22	404	336	337	336
   Total cold run time: 117907 ms
   Total hot run time: 41019 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4412	4202	4245	4202
   q2	385	276	265	265
   q3	2979	2768	2779	2768
   q4	1881	1613	1622	1613
   q5	5260	5269	5305	5269
   q6	213	123	124	123
   q7	2139	1784	1774	1774
   q8	3200	3395	3307	3307
   q9	8365	8395	8320	8320
   q10	3893	3621	3667	3621
   q11	582	481	488	481
   q12	786	590	604	590
   q13	16422	2987	2983	2983
   q14	290	277	260	260
   q15	516	481	482	481
   q16	461	401	428	401
   q17	1786	1471	1472	1471
   q18	7668	7734	7466	7466
   q19	1750	1621	1651	1621
   q20	1997	1787	1809	1787
   q21	4935	4673	4809	4673
   q22	605	525	545	525
   Total cold run time: 70525 ms
   Total hot run time: 54001 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids)disable unique key compute for mv [doris]

Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on code in PR #35898:
URL: https://github.com/apache/doris/pull/35898#discussion_r1628717517


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalOlapScan.java:
##########
@@ -455,4 +457,51 @@ public JSONObject toJson() {
         olapScan.put("Properties", properties);
         return olapScan;
     }
+
+    @Override
+    public void computeUnique(DataTrait.Builder builder) {
+        super.computeUnique(builder);
+        if (this.selectedIndexId != getTable().getBaseIndexId()) {
+            /*
+                computing unique doesn't work for mv, because mv's key may be different from base table
+                and the key can be any expression which is difficult to deduce if it's unique. for example
+                base table:
+                CREATE TABLE IF NOT EXISTS base(
+                    siteid INT(11) NOT NULL,
+                    citycode SMALLINT(6) NOT NULL,
+                    username VARCHAR(32) NOT NULL,
+                    pv BIGINT(20) SUM NOT NULL DEFAULT '0'
+                )
+                AGGREGATE KEY (siteid,citycode,username)
+                DISTRIBUTED BY HASH(siteid) BUCKETS 5 properties("replication_num" = "1");
+
+                case1:
+                create mv1:
+                create materialized view mv1 as select siteid, sum(pv) from base group by siteid;
+                the base table siteid + citycode + username is unique but the mv1's agg key siteid is not unique
+
+                case2:
+                create mv2:
+                create materialized view mv2 as select citycode * citycode, siteid, username from base;
+                the mv2's agg key citycode * citycode is not unique
+
+                for simplicity, we disable unique compute for mv
+             */
+            return;
+        }
+        Set<Slot> outputSet = Utils.fastToImmutableSet(getOutputSet());
+        if (getTable().getKeysType().isAggregationFamily()) {
+            ImmutableSet.Builder<Slot> uniqSlots = ImmutableSet.builderWithExpectedSize(outputSet.size());
+            for (Slot slot : outputSet) {
+                if (!(slot instanceof SlotReference)) {
+                    continue;
+                }
+                SlotReference slotRef = (SlotReference) slot;
+                if (slotRef.getColumn().isPresent() && slotRef.getColumn().get().isKey()) {
+                    uniqSlots.add(slot);
+                }
+            }
+            builder.addUniqueSlot(uniqSlots.build());
+        }
+    }

Review Comment:
   You should add the constraint code here, too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids)disable unique key compute for mv [doris]

Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on code in PR #35898:
URL: https://github.com/apache/doris/pull/35898#discussion_r1628714500


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java:
##########
@@ -128,20 +127,6 @@ public String qualifiedName() {
     @Override
     public void computeUnique(DataTrait.Builder builder) {
         Set<Slot> outputSet = Utils.fastToImmutableSet(getOutputSet());
-        if (table instanceof OlapTable && ((OlapTable) table).getKeysType().isAggregationFamily()) {
-            ImmutableSet.Builder<Slot> uniqSlots = ImmutableSet.builderWithExpectedSize(outputSet.size());
-            for (Slot slot : outputSet) {
-                if (!(slot instanceof SlotReference)) {
-                    continue;
-                }
-                SlotReference slotRef = (SlotReference) slot;
-                if (slotRef.getColumn().isPresent() && slotRef.getColumn().get().isKey()) {
-                    uniqSlots.add(slot);
-                }
-            }
-            builder.addUniqueSlot(uniqSlots.build());
-        }
-

Review Comment:
   Add a condition to check if it is a mv rather than remove those codes. They are useful if it's a trivial olapscan



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids)disable unique key compute for mv [doris]

Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on code in PR #35898:
URL: https://github.com/apache/doris/pull/35898#discussion_r1628717517


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalOlapScan.java:
##########
@@ -455,4 +457,51 @@ public JSONObject toJson() {
         olapScan.put("Properties", properties);
         return olapScan;
     }
+
+    @Override
+    public void computeUnique(DataTrait.Builder builder) {
+        super.computeUnique(builder);
+        if (this.selectedIndexId != getTable().getBaseIndexId()) {
+            /*
+                computing unique doesn't work for mv, because mv's key may be different from base table
+                and the key can be any expression which is difficult to deduce if it's unique. for example
+                base table:
+                CREATE TABLE IF NOT EXISTS base(
+                    siteid INT(11) NOT NULL,
+                    citycode SMALLINT(6) NOT NULL,
+                    username VARCHAR(32) NOT NULL,
+                    pv BIGINT(20) SUM NOT NULL DEFAULT '0'
+                )
+                AGGREGATE KEY (siteid,citycode,username)
+                DISTRIBUTED BY HASH(siteid) BUCKETS 5 properties("replication_num" = "1");
+
+                case1:
+                create mv1:
+                create materialized view mv1 as select siteid, sum(pv) from base group by siteid;
+                the base table siteid + citycode + username is unique but the mv1's agg key siteid is not unique
+
+                case2:
+                create mv2:
+                create materialized view mv2 as select citycode * citycode, siteid, username from base;
+                the mv2's agg key citycode * citycode is not unique
+
+                for simplicity, we disable unique compute for mv
+             */
+            return;
+        }
+        Set<Slot> outputSet = Utils.fastToImmutableSet(getOutputSet());
+        if (getTable().getKeysType().isAggregationFamily()) {
+            ImmutableSet.Builder<Slot> uniqSlots = ImmutableSet.builderWithExpectedSize(outputSet.size());
+            for (Slot slot : outputSet) {
+                if (!(slot instanceof SlotReference)) {
+                    continue;
+                }
+                SlotReference slotRef = (SlotReference) slot;
+                if (slotRef.getColumn().isPresent() && slotRef.getColumn().get().isKey()) {
+                    uniqSlots.add(slot);
+                }
+            }
+            builder.addUniqueSlot(uniqSlots.build());
+        }
+    }

Review Comment:
   You should add the constraint code here, too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids)disable unique key compute for mv [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2172805230

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 40528 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 94f4de5c91b98645a878ff189333f169da09df94, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	18473	4591	4384	4384
   q2	2627	193	205	193
   q3	10972	1120	1133	1120
   q4	10629	732	785	732
   q5	7558	2715	2616	2616
   q6	231	143	139	139
   q7	973	623	632	623
   q8	9623	2090	2059	2059
   q9	8918	6457	6433	6433
   q10	8941	3758	3698	3698
   q11	460	241	246	241
   q12	467	234	237	234
   q13	18795	2951	3014	2951
   q14	268	224	227	224
   q15	528	486	484	484
   q16	523	382	377	377
   q17	965	740	645	645
   q18	8011	7509	7464	7464
   q19	3242	1544	1471	1471
   q20	650	315	326	315
   q21	4958	3782	3897	3782
   q22	399	343	349	343
   Total cold run time: 118211 ms
   Total hot run time: 40528 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4446	4275	4228	4228
   q2	367	267	256	256
   q3	2990	2765	2721	2721
   q4	1854	1627	1660	1627
   q5	5237	5253	5240	5240
   q6	216	128	128	128
   q7	2158	1729	1665	1665
   q8	3186	3345	3297	3297
   q9	8373	8349	8343	8343
   q10	3881	3691	3624	3624
   q11	580	484	480	480
   q12	754	575	615	575
   q13	16420	2986	2985	2985
   q14	303	250	276	250
   q15	512	463	462	462
   q16	478	431	427	427
   q17	1757	1469	1462	1462
   q18	7595	7504	7433	7433
   q19	1973	1577	1596	1577
   q20	1985	1796	1763	1763
   q21	4839	4747	4735	4735
   q22	620	528	562	528
   Total cold run time: 70524 ms
   Total hot run time: 53806 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids)disable unique key compute for mv [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2175669109

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org