You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org> on 2024/03/20 10:15:09 UTC

[PR] [feature](merge-cloud) Add several cloud metrics [doris]

SWJTU-ZhangLei opened a new pull request, #32556:
URL: https://github.com/apache/doris/pull/32556

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several multiple dimensions cloud metrics [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2017771645

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit d2c22a6d12f8b88a2adb4b45f5941f5252da8107 with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       22.2 seconds inserted 10000000 Rows, about 450K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several multiple dimensions cloud metrics [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #32556:
URL: https://github.com/apache/doris/pull/32556


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several cloud metrics [doris]

Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2012089359

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several multiple dimensions cloud metrics [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2017149234

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit db8e17a26643208d87ef84b2221608f6baf24b34 with default session variables
   Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
   Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       20.7 seconds inserted 10000000 Rows, about 483K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several cloud metrics [doris]

Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2011658304

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several cloud metrics [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2011752035

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 180165 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit d875478d466f8b421e8d9209c18dc5187f337f89, data reload: false
   
   query1	932	358	333	333
   query2	7488	2011	2106	2011
   query3	6704	212	219	212
   query4	31176	20529	20500	20500
   query5	4341	388	502	388
   query6	268	174	171	171
   query7	4625	293	285	285
   query8	226	167	165	165
   query9	9749	2261	2208	2208
   query10	465	236	246	236
   query11	17176	14335	14229	14229
   query12	138	90	84	84
   query13	1630	414	399	399
   query14	10900	10799	10297	10297
   query15	264	211	191	191
   query16	7976	248	242	242
   query17	1968	541	543	541
   query18	1975	270	262	262
   query19	215	148	143	143
   query20	85	86	83	83
   query21	195	127	124	124
   query22	4653	4437	4520	4437
   query23	31911	30902	30841	30841
   query24	10822	2861	2800	2800
   query25	614	378	372	372
   query26	1214	148	150	148
   query27	2795	348	349	348
   query28	7918	1868	1881	1868
   query29	882	636	616	616
   query30	298	147	144	144
   query31	974	741	719	719
   query32	94	61	54	54
   query33	749	342	336	336
   query34	915	455	468	455
   query35	817	592	596	592
   query36	990	866	862	862
   query37	114	81	76	76
   query38	3501	3430	3412	3412
   query39	1548	1358	1489	1358
   query40	211	117	108	108
   query41	49	46	49	46
   query42	99	98	97	97
   query43	498	460	471	460
   query44	1155	710	714	710
   query45	281	264	259	259
   query46	1087	679	668	668
   query47	1669	1607	1595	1595
   query48	435	353	361	353
   query49	1137	330	334	330
   query50	751	373	356	356
   query51	6678	6678	6610	6610
   query52	107	95	85	85
   query53	345	270	275	270
   query54	300	249	258	249
   query55	86	86	80	80
   query56	266	243	220	220
   query57	1077	1015	1003	1003
   query58	232	203	213	203
   query59	2839	2572	2630	2572
   query60	273	243	252	243
   query61	115	111	110	110
   query62	596	400	406	400
   query63	302	274	271	271
   query64	5574	3985	3880	3880
   query65	3029	2991	2998	2991
   query66	871	371	355	355
   query67	14799	14291	14272	14272
   query68	8777	538	519	519
   query69	631	398	371	371
   query70	1234	1186	1173	1173
   query71	525	283	289	283
   query72	6379	2618	2500	2500
   query73	741	307	311	307
   query74	8286	6571	6625	6571
   query75	4631	2848	2900	2848
   query76	5301	847	913	847
   query77	685	258	248	248
   query78	10223	9770	9479	9479
   query79	8207	511	509	509
   query80	1515	380	373	373
   query81	513	210	229	210
   query82	847	205	212	205
   query83	184	140	137	137
   query84	282	78	78	78
   query85	1446	314	328	314
   query86	415	292	306	292
   query87	3746	3538	3537	3537
   query88	4982	2287	2275	2275
   query89	504	356	361	356
   query90	1997	174	176	174
   query91	164	130	130	130
   query92	63	47	47	47
   query93	6495	489	471	471
   query94	1107	172	170	170
   query95	422	324	321	321
   query96	599	269	270	269
   query97	3020	2869	2871	2869
   query98	226	213	198	198
   query99	1092	746	754	746
   Total cold run time: 309405 ms
   Total hot run time: 180165 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several multiple dimensions cloud metrics [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2017733969

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38504 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit d2c22a6d12f8b88a2adb4b45f5941f5252da8107, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17634	4463	4206	4206
   q2	2532	176	173	173
   q3	11370	1121	1178	1121
   q4	10896	825	858	825
   q5	7589	3066	3085	3066
   q6	212	127	126	126
   q7	1065	620	614	614
   q8	9517	2059	2047	2047
   q9	7341	6761	6737	6737
   q10	8751	3467	3569	3467
   q11	438	229	217	217
   q12	366	203	195	195
   q13	17819	2841	2862	2841
   q14	222	201	206	201
   q15	514	448	451	448
   q16	492	374	373	373
   q17	979	587	608	587
   q18	7220	6455	6542	6455
   q19	3397	1380	1447	1380
   q20	562	272	249	249
   q21	3588	2871	3040	2871
   q22	359	312	305	305
   Total cold run time: 112863 ms
   Total hot run time: 38504 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4105	4077	4049	4049
   q2	327	243	241	241
   q3	3066	2914	2850	2850
   q4	1830	1606	1532	1532
   q5	5372	5451	5342	5342
   q6	198	117	117	117
   q7	2231	1891	1879	1879
   q8	3169	3299	3301	3299
   q9	8727	8738	8709	8709
   q10	3838	3762	3816	3762
   q11	534	440	452	440
   q12	730	531	530	530
   q13	16926	2896	2865	2865
   q14	272	239	267	239
   q15	501	457	462	457
   q16	489	421	409	409
   q17	1737	1505	1475	1475
   q18	7405	7155	7207	7155
   q19	1735	1527	1537	1527
   q20	1895	1719	1720	1719
   q21	4793	4689	4944	4689
   q22	514	427	447	427
   Total cold run time: 70394 ms
   Total hot run time: 53712 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several cloud metrics [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2012162860

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 181426 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit d255d816b72a7f1d0b275d255d866917191f20dd, data reload: false
   
   query1	919	361	335	335
   query2	7361	2045	1996	1996
   query3	6709	211	208	208
   query4	31324	20657	20632	20632
   query5	4550	401	482	401
   query6	268	172	173	172
   query7	4618	299	285	285
   query8	236	177	174	174
   query9	9953	2286	2280	2280
   query10	461	247	274	247
   query11	14711	14253	14108	14108
   query12	145	92	86	86
   query13	1632	421	421	421
   query14	11866	11288	11173	11173
   query15	269	206	199	199
   query16	8179	264	261	261
   query17	1958	586	542	542
   query18	2099	286	279	279
   query19	332	156	168	156
   query20	92	90	92	90
   query21	202	129	127	127
   query22	4653	4448	4413	4413
   query23	31804	31028	31247	31028
   query24	11519	2819	2783	2783
   query25	652	398	386	386
   query26	1762	157	159	157
   query27	2961	344	345	344
   query28	8037	1927	1898	1898
   query29	1015	673	621	621
   query30	307	148	147	147
   query31	963	724	721	721
   query32	98	59	56	56
   query33	772	268	266	266
   query34	1035	477	473	473
   query35	830	607	613	607
   query36	1005	880	868	868
   query37	156	78	77	77
   query38	3583	3423	3431	3423
   query39	1469	1364	1370	1364
   query40	296	119	117	117
   query41	53	47	49	47
   query42	109	95	103	95
   query43	484	448	447	447
   query44	1138	722	703	703
   query45	260	261	263	261
   query46	1099	680	678	678
   query47	1676	1600	1602	1600
   query48	438	365	358	358
   query49	1224	355	337	337
   query50	757	369	379	369
   query51	6702	6645	6604	6604
   query52	114	90	100	90
   query53	342	282	274	274
   query54	320	256	250	250
   query55	85	79	82	79
   query56	266	313	237	237
   query57	1115	1017	1018	1017
   query58	229	208	213	208
   query59	2714	2501	2625	2501
   query60	283	252	268	252
   query61	120	120	118	118
   query62	615	401	387	387
   query63	300	282	287	282
   query64	6814	3827	3629	3629
   query65	3138	3027	3031	3027
   query66	1402	346	336	336
   query67	15065	14348	14286	14286
   query68	7771	520	525	520
   query69	612	373	396	373
   query70	1270	1181	1132	1132
   query71	523	277	280	277
   query72	6529	2670	2544	2544
   query73	749	313	320	313
   query74	7649	6585	6608	6585
   query75	4311	2892	2837	2837
   query76	4977	912	911	911
   query77	647	271	250	250
   query78	10456	9643	9525	9525
   query79	10123	512	525	512
   query80	1721	388	389	388
   query81	523	207	209	207
   query82	883	203	204	203
   query83	216	143	148	143
   query84	284	78	81	78
   query85	1517	325	311	311
   query86	476	305	316	305
   query87	3738	3540	3534	3534
   query88	5091	2307	2302	2302
   query89	503	364	366	364
   query90	1994	175	176	175
   query91	161	129	131	129
   query92	59	46	47	46
   query93	7103	509	481	481
   query94	1239	178	173	173
   query95	424	327	325	325
   query96	616	270	274	270
   query97	3067	2871	2872	2871
   query98	250	205	203	203
   query99	1241	769	746	746
   Total cold run time: 314386 ms
   Total hot run time: 181426 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several cloud metrics [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2012170965

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit d255d816b72a7f1d0b275d255d866917191f20dd with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
   Insert into select:       22.5 seconds inserted 10000000 Rows, about 444K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several cloud metrics [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2012135493

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38183 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit d255d816b72a7f1d0b275d255d866917191f20dd, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17616	4251	4130	4130
   q2	2023	161	149	149
   q3	10942	1209	1236	1209
   q4	10575	794	767	767
   q5	7544	3056	3036	3036
   q6	205	126	126	126
   q7	1069	652	584	584
   q8	10278	2060	2025	2025
   q9	7219	6599	6555	6555
   q10	8991	3466	3592	3466
   q11	548	226	220	220
   q12	644	211	205	205
   q13	19036	2872	2874	2872
   q14	225	199	210	199
   q15	502	452	456	452
   q16	456	354	354	354
   q17	959	515	645	515
   q18	7224	6515	6502	6502
   q19	2807	1437	1462	1437
   q20	515	258	246	246
   q21	3589	2975	2839	2839
   q22	352	295	296	295
   Total cold run time: 113319 ms
   Total hot run time: 38183 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4172	4080	4095	4080
   q2	333	241	227	227
   q3	3014	2866	2844	2844
   q4	1814	1587	1540	1540
   q5	5216	5264	5281	5264
   q6	199	117	119	117
   q7	2226	1862	1873	1862
   q8	3167	3272	3273	3272
   q9	8564	8545	8525	8525
   q10	3704	3715	3684	3684
   q11	535	434	434	434
   q12	718	603	558	558
   q13	16889	2859	2841	2841
   q14	272	255	273	255
   q15	469	447	456	447
   q16	459	418	407	407
   q17	1733	1476	1465	1465
   q18	7419	7144	7138	7138
   q19	1612	1562	1527	1527
   q20	1887	1698	1705	1698
   q21	4738	4655	4771	4655
   q22	510	448	466	448
   Total cold run time: 69650 ms
   Total hot run time: 53288 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several multiple dimensions cloud metrics [doris]

Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2017703487

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several multiple dimensions cloud metrics [doris]

Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2017126001

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several cloud metrics [doris]

Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2009435042

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several cloud metrics [doris]

Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2011056902

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several cloud metrics [doris]

Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2011662782

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several multiple dimensions cloud metrics [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2017760484

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 186110 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit d2c22a6d12f8b88a2adb4b45f5941f5252da8107, data reload: false
   
   query1	944	375	359	359
   query2	7541	1915	1939	1915
   query3	6712	212	209	209
   query4	31489	21601	21277	21277
   query5	4343	403	403	403
   query6	265	182	184	182
   query7	4627	295	293	293
   query8	224	188	177	177
   query9	9150	2293	2253	2253
   query10	569	251	252	251
   query11	14665	14318	14188	14188
   query12	144	93	90	90
   query13	1627	439	425	425
   query14	12300	11504	11348	11348
   query15	252	216	196	196
   query16	8022	273	257	257
   query17	1949	592	556	556
   query18	2053	295	287	287
   query19	252	162	159	159
   query20	94	86	91	86
   query21	199	129	134	129
   query22	5005	4817	4786	4786
   query23	33390	32480	32543	32480
   query24	11683	2828	2836	2828
   query25	655	388	381	381
   query26	1793	153	150	150
   query27	3017	351	350	350
   query28	7600	1868	1825	1825
   query29	1037	645	609	609
   query30	303	146	145	145
   query31	949	719	730	719
   query32	92	55	56	55
   query33	770	241	247	241
   query34	1068	485	486	485
   query35	832	613	614	613
   query36	1016	884	893	884
   query37	173	74	75	74
   query38	3591	3467	3398	3398
   query39	1456	1436	1404	1404
   query40	291	110	111	110
   query41	49	46	46	46
   query42	102	94	95	94
   query43	474	441	435	435
   query44	1227	732	718	718
   query45	275	252	255	252
   query46	1104	689	699	689
   query47	1914	1863	1848	1848
   query48	459	351	347	347
   query49	1214	330	339	330
   query50	750	371	372	371
   query51	6686	6714	6591	6591
   query52	116	84	98	84
   query53	345	276	275	275
   query54	311	234	242	234
   query55	85	78	80	78
   query56	235	226	220	220
   query57	1201	1131	1141	1131
   query58	227	204	207	204
   query59	2692	2617	2616	2616
   query60	260	242	241	241
   query61	99	102	120	102
   query62	659	445	445	445
   query63	314	283	275	275
   query64	6887	3964	4092	3964
   query65	3132	3064	3072	3064
   query66	1427	383	374	374
   query67	15405	14796	15020	14796
   query68	5322	511	521	511
   query69	566	385	387	385
   query70	1181	1159	1130	1130
   query71	420	274	273	273
   query72	6891	2852	2675	2675
   query73	706	316	318	316
   query74	7773	6363	6447	6363
   query75	3517	2849	2839	2839
   query76	3821	877	890	877
   query77	599	274	257	257
   query78	10915	10225	10220	10220
   query79	8603	528	521	521
   query80	1889	374	373	373
   query81	544	210	217	210
   query82	1173	213	210	210
   query83	321	144	177	144
   query84	289	81	78	78
   query85	1674	322	318	318
   query86	487	291	277	277
   query87	3769	3550	3565	3550
   query88	5029	2278	2269	2269
   query89	542	373	375	373
   query90	1973	175	172	172
   query91	167	134	140	134
   query92	59	47	46	46
   query93	6341	498	471	471
   query94	1234	173	172	172
   query95	427	331	328	328
   query96	595	271	266	266
   query97	3089	2869	2867	2867
   query98	223	218	205	205
   query99	1180	879	899	879
   Total cold run time: 311110 ms
   Total hot run time: 186110 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several multiple dimensions cloud metrics [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2018227560

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several multiple dimensions cloud metrics [doris]

Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #32556:
URL: https://github.com/apache/doris/pull/32556#discussion_r1537557049


##########
fe/fe-core/src/main/java/org/apache/doris/metric/MetricCalculator.java:
##########
@@ -77,4 +87,69 @@ private void update() {
         }
         MetricRepo.GAUGE_MAX_TABLET_COMPACTION_SCORE.setValue(maxCompactionScore);
     }
+
+    private void initCloudMetrics() {
+        if (!Config.isCloudMode()) {

Review Comment:
   > ditto
   resolved



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several cloud metrics [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2011728105

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 37360 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit d875478d466f8b421e8d9209c18dc5187f337f89, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17666	4220	4055	4055
   q2	2018	151	156	151
   q3	10587	1119	1183	1119
   q4	10231	784	808	784
   q5	7462	2948	2912	2912
   q6	202	123	124	123
   q7	1043	574	569	569
   q8	9362	1978	1970	1970
   q9	7087	6491	6399	6399
   q10	8403	3313	3418	3313
   q11	433	219	220	219
   q12	408	208	201	201
   q13	17791	2852	2861	2852
   q14	226	220	215	215
   q15	508	457	458	457
   q16	455	358	355	355
   q17	946	582	606	582
   q18	7144	6372	6349	6349
   q19	5215	1380	1442	1380
   q20	538	263	246	246
   q21	3534	2879	2814	2814
   q22	366	295	318	295
   Total cold run time: 111625 ms
   Total hot run time: 37360 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4144	4078	4083	4078
   q2	323	226	242	226
   q3	2946	2850	2880	2850
   q4	1843	1501	1532	1501
   q5	5220	5233	5231	5231
   q6	194	119	119	119
   q7	2220	1883	1829	1829
   q8	3138	3267	3260	3260
   q9	8590	8513	8520	8513
   q10	3709	3652	3615	3615
   q11	535	427	443	427
   q12	744	513	553	513
   q13	16938	2860	2858	2858
   q14	279	247	272	247
   q15	497	438	448	438
   q16	456	413	418	413
   q17	1727	1467	1443	1443
   q18	7403	7055	7050	7050
   q19	2243	1508	1504	1504
   q20	1903	1686	1681	1681
   q21	4788	4781	4783	4781
   q22	527	439	442	439
   Total cold run time: 70367 ms
   Total hot run time: 53016 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several cloud metrics [doris]

Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2009202122

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several cloud metrics [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2009201353

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
   See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several multiple dimensions cloud metrics [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2017146976

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 185981 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit db8e17a26643208d87ef84b2221608f6baf24b34, data reload: false
   
   query1	932	371	350	350
   query2	7377	2019	1992	1992
   query3	6704	216	222	216
   query4	31773	21285	21371	21285
   query5	4324	462	424	424
   query6	291	171	172	171
   query7	4621	292	286	286
   query8	240	163	172	163
   query9	9201	2251	2250	2250
   query10	559	245	248	245
   query11	17494	14408	14484	14408
   query12	133	90	84	84
   query13	1625	409	414	409
   query14	11160	11445	10669	10669
   query15	287	207	196	196
   query16	8151	259	257	257
   query17	1951	595	564	564
   query18	2080	294	290	290
   query19	300	160	160	160
   query20	102	89	92	89
   query21	217	130	129	129
   query22	5014	4750	4741	4741
   query23	33691	32585	32928	32585
   query24	10713	2830	2833	2830
   query25	600	386	397	386
   query26	781	153	159	153
   query27	2432	354	352	352
   query28	6685	1894	1843	1843
   query29	917	667	592	592
   query30	300	149	150	149
   query31	985	731	724	724
   query32	94	58	56	56
   query33	772	257	252	252
   query34	1103	489	495	489
   query35	814	604	608	604
   query36	1016	866	855	855
   query37	118	75	76	75
   query38	3601	3399	3452	3399
   query39	1489	1422	1417	1417
   query40	218	110	109	109
   query41	49	46	44	44
   query42	102	93	95	93
   query43	466	425	451	425
   query44	1181	723	714	714
   query45	275	242	264	242
   query46	1118	702	703	702
   query47	1918	1831	1860	1831
   query48	432	351	351	351
   query49	1099	334	330	330
   query50	751	376	369	369
   query51	6738	6634	6627	6627
   query52	105	86	98	86
   query53	340	276	274	274
   query54	307	249	241	241
   query55	83	78	80	78
   query56	254	224	223	223
   query57	1211	1132	1137	1132
   query58	237	198	206	198
   query59	2745	2627	2546	2546
   query60	256	237	248	237
   query61	96	96	96	96
   query62	668	442	463	442
   query63	308	282	274	274
   query64	5579	4020	4121	4020
   query65	3087	3009	3039	3009
   query66	883	383	382	382
   query67	15292	14924	14731	14731
   query68	5254	522	519	519
   query69	566	380	385	380
   query70	1254	1189	1181	1181
   query71	425	282	287	282
   query72	6450	2876	2703	2703
   query73	697	312	326	312
   query74	7916	6767	6655	6655
   query75	3557	2879	2798	2798
   query76	3701	878	869	869
   query77	625	275	278	275
   query78	10826	10161	10240	10161
   query79	8739	513	517	513
   query80	2010	381	374	374
   query81	555	212	220	212
   query82	1417	203	193	193
   query83	315	142	141	141
   query84	287	79	80	79
   query85	1720	332	309	309
   query86	492	297	301	297
   query87	3777	3542	3542	3542
   query88	5081	2282	2264	2264
   query89	559	373	363	363
   query90	1957	185	177	177
   query91	168	138	137	137
   query92	59	47	47	47
   query93	6660	484	480	480
   query94	1232	177	172	172
   query95	426	336	325	325
   query96	605	268	275	268
   query97	3075	2879	2925	2879
   query98	228	222	216	216
   query99	1232	921	913	913
   Total cold run time: 308310 ms
   Total hot run time: 185981 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several multiple dimensions cloud metrics [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2017138419

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38459 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit db8e17a26643208d87ef84b2221608f6baf24b34, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	18026	6022	4298	4298
   q2	2473	166	155	155
   q3	11700	1138	1265	1138
   q4	10716	753	755	753
   q5	7846	3166	3117	3117
   q6	210	127	123	123
   q7	1053	610	578	578
   q8	9641	1999	2041	1999
   q9	7403	6765	6779	6765
   q10	9144	3450	3543	3450
   q11	425	223	231	223
   q12	379	200	199	199
   q13	17807	2898	2853	2853
   q14	245	201	199	199
   q15	501	458	462	458
   q16	498	385	373	373
   q17	959	534	587	534
   q18	7223	6515	6433	6433
   q19	1858	1399	1427	1399
   q20	550	252	249	249
   q21	3566	2867	3018	2867
   q22	355	296	297	296
   Total cold run time: 112578 ms
   Total hot run time: 38459 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4179	4172	4094	4094
   q2	327	237	229	229
   q3	2990	2879	2864	2864
   q4	1859	1571	1614	1571
   q5	5288	5361	5341	5341
   q6	194	118	117	117
   q7	2241	1859	1817	1817
   q8	3173	3329	3325	3325
   q9	8726	8672	8765	8672
   q10	3785	3774	3765	3765
   q11	552	441	443	441
   q12	710	569	549	549
   q13	16898	2870	2881	2870
   q14	284	252	259	252
   q15	500	466	462	462
   q16	473	417	433	417
   q17	1759	1520	1490	1490
   q18	7596	7182	7232	7182
   q19	1633	1571	1569	1569
   q20	1901	1728	1735	1728
   q21	4938	4835	4712	4712
   q22	540	454	474	454
   Total cold run time: 70546 ms
   Total hot run time: 53921 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several multiple dimensions cloud metrics [doris]

Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #32556:
URL: https://github.com/apache/doris/pull/32556#discussion_r1537240582


##########
fe/fe-core/src/main/java/org/apache/doris/metric/CloudMetrics.java:
##########
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.metric;
+
+import org.apache.doris.common.Config;
+import org.apache.doris.metric.Metric.MetricUnit;
+
+import com.codahale.metrics.Histogram;
+import com.codahale.metrics.MetricRegistry;
+
+public class CloudMetrics {
+    protected static String CLOUD_CLUSTER_DELIMITER = "@delimiter#";
+    protected static AutoMappedMetric<LongCounterMetric> CLUSTER_REQUEST_ALL_COUNTER;
+    protected static AutoMappedMetric<LongCounterMetric> CLUSTER_QUERY_ALL_COUNTER;
+    protected static AutoMappedMetric<LongCounterMetric> CLUSTER_QUERY_ERR_COUNTER;
+
+    protected static AutoMappedMetric<GaugeMetricImpl<Double>> CLUSTER_REQUEST_PER_SECOND_GAUGE;
+    protected static AutoMappedMetric<GaugeMetricImpl<Double>> CLUSTER_QUERY_PER_SECOND_GAUGE;
+    protected static AutoMappedMetric<GaugeMetricImpl<Double>> CLUSTER_QUERY_ERR_RATE_GAUGE;
+
+    protected static AutoMappedMetric<Histogram> CLUSTER_QUERY_LATENCY_HISTO;
+
+    protected static AutoMappedMetric<GaugeMetricImpl<Integer>> CLUSTER_BACKEND_ALIVE;
+    protected static AutoMappedMetric<GaugeMetricImpl<Integer>> CLUSTER_BACKEND_ALIVE_TOTAL;
+
+    protected static void init() {
+        if (!Config.isCloudMode()) {

Review Comment:
   Config.isNotCloudMode()



##########
fe/fe-core/src/main/java/org/apache/doris/metric/MetricCalculator.java:
##########
@@ -77,4 +87,69 @@ private void update() {
         }
         MetricRepo.GAUGE_MAX_TABLET_COMPACTION_SCORE.setValue(maxCompactionScore);
     }
+
+    private void initCloudMetrics() {
+        if (!Config.isCloudMode()) {

Review Comment:
   ditto
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](merge-cloud) Add several multiple dimensions cloud metrics [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32556:
URL: https://github.com/apache/doris/pull/32556#issuecomment-2020543678

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org