You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "hqx871 (via GitHub)" <gi...@apache.org> on 2024/06/16 05:49:13 UTC

[PR] [feat](agg) support second key [doris]

hqx871 opened a new pull request, #36352:
URL: https://github.com/apache/doris/pull/36352

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   For uniq and agg table, Sometimes use will compute md5 value of some key columns and put md5 as key while put those key as value. For example
   - when key column is too many, using md5 as key will reduce readed key columns as agg/uniq table will need to read all the key columns although user do not select those key columns
   - use may need to use array as key, but current doris does not support
   
   To improve those cases, I add a table property 'second_key' to let ScanNode to push predicates of those key columns to rowset.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feat](agg) support second key [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171256472

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 173211 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit efaf0c4f484996fd691b4aff2c2508b0c382c823, data reload: false
   
   query1	911	385	377	377
   query2	6456	2509	2376	2376
   query3	6628	204	212	204
   query4	22261	17308	17254	17254
   query5	3636	476	446	446
   query6	248	158	149	149
   query7	4647	297	300	297
   query8	318	289	284	284
   query9	8469	2368	2336	2336
   query10	573	291	273	273
   query11	10703	10043	10083	10043
   query12	113	86	83	83
   query13	1630	365	359	359
   query14	9164	6710	6790	6710
   query15	230	186	187	186
   query16	7222	263	272	263
   query17	1404	568	530	530
   query18	1917	282	278	278
   query19	197	163	159	159
   query20	90	85	83	83
   query21	218	131	123	123
   query22	4394	3960	4032	3960
   query23	33772	33675	33566	33566
   query24	11394	2836	2864	2836
   query25	624	372	367	367
   query26	1283	162	164	162
   query27	2390	327	326	326
   query28	7215	2120	2107	2107
   query29	898	654	610	610
   query30	251	153	154	153
   query31	993	775	779	775
   query32	86	56	52	52
   query33	753	288	304	288
   query34	1031	502	472	472
   query35	783	676	651	651
   query36	1174	988	964	964
   query37	151	84	73	73
   query38	2994	2837	2805	2805
   query39	921	779	817	779
   query40	217	127	125	125
   query41	54	49	52	49
   query42	117	94	97	94
   query43	566	553	538	538
   query44	1225	713	732	713
   query45	192	159	166	159
   query46	1076	740	710	710
   query47	1858	1782	1763	1763
   query48	364	300	289	289
   query49	854	405	392	392
   query50	770	383	399	383
   query51	6791	6719	6680	6680
   query52	98	90	102	90
   query53	357	284	295	284
   query54	880	448	439	439
   query55	75	75	71	71
   query56	274	250	260	250
   query57	1123	1046	1034	1034
   query58	245	256	257	256
   query59	3544	3340	3201	3201
   query60	286	261	269	261
   query61	94	94	103	94
   query62	637	441	433	433
   query63	311	283	290	283
   query64	8897	2295	1775	1775
   query65	3170	3151	3108	3108
   query66	884	326	334	326
   query67	15513	15064	14964	14964
   query68	4564	530	534	530
   query69	598	478	395	395
   query70	1199	1152	1095	1095
   query71	416	267	262	262
   query72	8393	5359	6028	5359
   query73	776	324	324	324
   query74	5969	5559	5463	5463
   query75	3386	2632	2618	2618
   query76	2717	910	945	910
   query77	646	308	300	300
   query78	10475	9880	9829	9829
   query79	2486	500	516	500
   query80	979	469	482	469
   query81	601	219	231	219
   query82	1361	98	102	98
   query83	263	166	160	160
   query84	239	90	83	83
   query85	1405	289	331	289
   query86	454	320	325	320
   query87	3258	3081	3125	3081
   query88	4312	2354	2340	2340
   query89	475	394	386	386
   query90	1695	188	192	188
   query91	124	97	94	94
   query92	59	47	46	46
   query93	2350	502	498	498
   query94	1040	186	183	183
   query95	394	312	306	306
   query96	597	265	268	265
   query97	3308	3052	3043	3043
   query98	220	200	194	194
   query99	1207	854	841	841
   Total cold run time: 274580 ms
   Total hot run time: 173211 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feat](agg) support second key [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171064712

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feat](agg) support second key [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171266367

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.08 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit efaf0c4f484996fd691b4aff2c2508b0c382c823, data reload: false
   
   query1	0.04	0.03	0.03
   query2	0.07	0.03	0.04
   query3	0.23	0.04	0.05
   query4	1.67	0.08	0.08
   query5	0.49	0.47	0.50
   query6	1.13	0.72	0.72
   query7	0.02	0.01	0.02
   query8	0.05	0.04	0.04
   query9	0.55	0.49	0.49
   query10	0.55	0.56	0.54
   query11	0.15	0.11	0.11
   query12	0.15	0.12	0.12
   query13	0.60	0.59	0.60
   query14	0.78	0.78	0.77
   query15	0.84	0.82	0.81
   query16	0.34	0.37	0.37
   query17	1.04	1.02	0.98
   query18	0.22	0.26	0.23
   query19	1.80	1.65	1.75
   query20	0.02	0.02	0.01
   query21	15.40	0.65	0.65
   query22	4.00	7.70	1.46
   query23	18.29	1.31	1.36
   query24	2.10	0.23	0.22
   query25	0.15	0.08	0.09
   query26	0.27	0.17	0.18
   query27	0.08	0.07	0.08
   query28	13.17	1.02	0.99
   query29	12.64	3.22	3.24
   query30	0.26	0.07	0.07
   query31	2.84	0.38	0.38
   query32	3.26	0.48	0.47
   query33	2.91	2.87	2.87
   query34	17.04	4.54	4.49
   query35	4.47	4.51	4.48
   query36	0.66	0.46	0.47
   query37	0.18	0.15	0.15
   query38	0.15	0.15	0.14
   query39	0.05	0.03	0.03
   query40	0.17	0.15	0.15
   query41	0.10	0.05	0.05
   query42	0.06	0.05	0.05
   query43	0.04	0.03	0.04
   Total cold run time: 109.03 s
   Total hot run time: 30.08 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feat](agg) support second key [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171230655

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 40774 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit efaf0c4f484996fd691b4aff2c2508b0c382c823, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17600	4432	4302	4302
   q2	2020	196	198	196
   q3	10448	1191	1138	1138
   q4	10200	811	766	766
   q5	7481	2689	2665	2665
   q6	221	138	140	138
   q7	962	619	610	610
   q8	9223	2106	2122	2106
   q9	8856	6561	6551	6551
   q10	8959	3731	3710	3710
   q11	462	235	239	235
   q12	486	234	229	229
   q13	17979	3004	2988	2988
   q14	257	223	220	220
   q15	518	489	490	489
   q16	515	373	373	373
   q17	989	681	747	681
   q18	8144	7517	7331	7331
   q19	5938	1554	1575	1554
   q20	663	298	316	298
   q21	5023	3863	3872	3863
   q22	394	332	331	331
   Total cold run time: 117338 ms
   Total hot run time: 40774 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4407	4279	4236	4236
   q2	372	269	262	262
   q3	2975	3031	2936	2936
   q4	1966	1655	1768	1655
   q5	5554	5515	5510	5510
   q6	229	134	138	134
   q7	2210	1895	1843	1843
   q8	3335	3404	3523	3404
   q9	8773	8732	8887	8732
   q10	4070	3934	3768	3768
   q11	579	498	512	498
   q12	818	648	614	614
   q13	17001	3171	3188	3171
   q14	288	290	274	274
   q15	537	495	476	476
   q16	482	421	429	421
   q17	1853	1542	1520	1520
   q18	8095	7922	7794	7794
   q19	1851	1658	1715	1658
   q20	3124	1929	1842	1842
   q21	5312	5095	4870	4870
   q22	642	535	558	535
   Total cold run time: 74473 ms
   Total hot run time: 56153 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feat](agg) support second key [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171062724

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
   See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feat](agg) support second key [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171068594

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feat](agg) support second key [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171266909

   TeamCity be ut coverage result:
    Function Coverage: 36.47% (9001/24678) 
    Line Coverage: 28.01% (73716/263133)
    Region Coverage: 27.49% (38295/139319)
    Branch Coverage: 24.18% (19520/80716)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/efaf0c4f484996fd691b4aff2c2508b0c382c823_efaf0c4f484996fd691b4aff2c2508b0c382c823/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feat](agg) support second key [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #36352: [feat](agg) support second key
URL: https://github.com/apache/doris/pull/36352


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feat](agg) support second key [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171683228

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feat](agg) support second key [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2543384965

   We're closing this PR because it hasn't been updated in a while.
   This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
   If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feat](agg) support second key [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171116046

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feat](agg) support second key [doris]

Posted by "hqx871 (via GitHub)" <gi...@apache.org>.
hqx871 commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171118735

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org