You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Jibing-Li (via GitHub)" <gi...@apache.org> on 2024/03/07 14:31:10 UTC

[PR] [fix](statistics)Add synchronize for modify analysisTaskInfoMap and analysisJobInfoMap. #31940 [doris]

Jibing-Li opened a new pull request, #31967:
URL: https://github.com/apache/doris/pull/31967

   backport https://github.com/apache/doris/pull/31940
   
   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](statistics)Add synchronize for modify analysisTaskInfoMap and analysisJobInfoMap. #31940 [doris]

Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li merged PR #31967:
URL: https://github.com/apache/doris/pull/31967


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](statistics)Add synchronize for modify analysisTaskInfoMap and analysisJobInfoMap. #31940 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31967:
URL: https://github.com/apache/doris/pull/31967#issuecomment-1983682686

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 50015 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit a3e1f441e79b9089f4f553c92dea902e32a3b739, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17929	4386	4361	4361
   q2	2047	154	138	138
   q3	10505	1901	1926	1901
   q4	10289	1274	1332	1274
   q5	8525	3951	3945	3945
   q6	234	123	124	123
   q7	2023	1589	1605	1589
   q8	9263	2686	2687	2686
   q9	11007	10595	10416	10416
   q10	8622	3508	3529	3508
   q11	404	239	242	239
   q12	466	294	302	294
   q13	18368	4032	4107	4032
   q14	356	325	333	325
   q15	507	460	463	460
   q16	722	624	622	622
   q17	1136	993	997	993
   q18	7291	6922	6878	6878
   q19	1677	1567	1472	1472
   q20	539	296	285	285
   q21	4453	4080	4107	4080
   q22	515	398	394	394
   Total cold run time: 116878 ms
   Total hot run time: 50015 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4377	4311	4297	4297
   q2	320	219	225	219
   q3	4144	4120	4112	4112
   q4	2745	2748	2724	2724
   q5	7245	7185	7141	7141
   q6	239	119	120	119
   q7	3235	2804	2824	2804
   q8	4337	4450	4443	4443
   q9	17081	16993	17061	16993
   q10	4240	4270	4256	4256
   q11	750	692	701	692
   q12	1005	861	847	847
   q13	6737	3750	3757	3750
   q14	442	421	428	421
   q15	498	465	451	451
   q16	744	706	697	697
   q17	3863	3895	3801	3801
   q18	8679	8803	8768	8768
   q19	1689	1692	1671	1671
   q20	2408	2147	2088	2088
   q21	8464	8393	8525	8393
   q22	1017	962	920	920
   Total cold run time: 84259 ms
   Total hot run time: 79607 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](statistics)Add synchronize for modify analysisTaskInfoMap and analysisJobInfoMap. #31940 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31967:
URL: https://github.com/apache/doris/pull/31967#issuecomment-1983716217

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 201954 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit a3e1f441e79b9089f4f553c92dea902e32a3b739, data reload: false
   
   query1	912	389	379	379
   query2	6520	2156	2178	2156
   query3	6917	192	195	192
   query4	20238	17853	18550	17853
   query5	19706	6549	6493	6493
   query6	292	214	223	214
   query7	4169	292	307	292
   query8	232	221	231	221
   query9	3073	2655	2556	2556
   query10	404	292	296	292
   query11	11408	10781	10633	10633
   query12	120	70	72	70
   query13	5572	630	637	630
   query14	17909	13437	13792	13437
   query15	364	241	242	241
   query16	6474	265	258	258
   query17	1772	1449	895	895
   query18	2291	402	406	402
   query19	219	149	150	149
   query20	75	76	75	75
   query21	188	92	95	92
   query22	5215	5240	5161	5161
   query23	32795	32014	32564	32014
   query24	7002	6573	6658	6573
   query25	564	466	448	448
   query26	1292	179	168	168
   query27	2618	316	304	304
   query28	7268	2206	2187	2187
   query29	2881	2781	2633	2633
   query30	254	164	167	164
   query31	1209	726	703	703
   query32	66	58	58	58
   query33	426	246	254	246
   query34	850	461	471	461
   query35	1090	927	917	917
   query36	1511	1256	1333	1256
   query37	87	60	60	60
   query38	3077	2893	2920	2893
   query39	1373	1333	1294	1294
   query40	203	89	94	89
   query41	34	32	31	31
   query42	85	79	82	79
   query43	608	564	502	502
   query44	1105	710	712	710
   query45	242	227	229	227
   query46	1225	980	957	957
   query47	1804	1692	1716	1692
   query48	970	667	667	667
   query49	646	374	350	350
   query50	858	630	615	615
   query51	4686	4655	4704	4655
   query52	90	78	81	78
   query53	450	316	325	316
   query54	2659	2469	2441	2441
   query55	82	77	78	77
   query56	206	206	190	190
   query57	1294	1106	1097	1097
   query58	206	201	186	186
   query59	3415	3341	3167	3167
   query60	207	195	202	195
   query61	85	82	86	82
   query62	778	552	574	552
   query63	480	335	331	331
   query64	2414	1496	1314	1314
   query65	3613	3509	3548	3509
   query66	813	374	371	371
   query67	15586	15885	15876	15876
   query68	10534	648	669	648
   query69	573	345	333	333
   query70	2480	1704	1616	1616
   query71	426	305	309	305
   query72	6424	3438	3436	3436
   query73	728	321	319	319
   query74	6367	5874	5792	5792
   query75	5446	3748	3653	3653
   query76	6666	1154	1199	1154
   query77	1178	259	250	250
   query78	12705	11866	11399	11399
   query79	11287	651	627	627
   query80	1097	385	378	378
   query81	488	229	233	229
   query82	698	94	97	94
   query83	170	131	129	129
   query84	257	69	66	66
   query85	899	276	276	276
   query86	393	364	357	357
   query87	3246	3002	3063	3002
   query88	5167	2288	2291	2288
   query89	432	328	271	271
   query90	2085	211	191	191
   query91	157	113	112	112
   query92	57	50	50	50
   query93	6414	544	557	544
   query94	763	203	206	203
   query95	1095	1061	1047	1047
   query96	636	317	320	317
   query97	6460	6390	6377	6377
   query98	189	179	168	168
   query99	3140	897	942	897
   Total cold run time: 321934 ms
   Total hot run time: 201954 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](statistics)Add synchronize for modify analysisTaskInfoMap and analysisJobInfoMap. #31940 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31967:
URL: https://github.com/apache/doris/pull/31967#issuecomment-1983628327

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](statistics)Add synchronize for modify analysisTaskInfoMap and analysisJobInfoMap. #31940 [doris]

Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #31967:
URL: https://github.com/apache/doris/pull/31967#issuecomment-1983628791

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](statistics)Add synchronize for modify analysisTaskInfoMap and analysisJobInfoMap. #31940 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31967:
URL: https://github.com/apache/doris/pull/31967#issuecomment-1983728382

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.25 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit a3e1f441e79b9089f4f553c92dea902e32a3b739, data reload: false
   
   query1	0.02	0.02	0.02
   query2	0.07	0.02	0.02
   query3	0.25	0.05	0.04
   query4	1.80	0.06	0.06
   query5	0.53	0.53	0.51
   query6	1.25	0.62	0.67
   query7	0.01	0.01	0.01
   query8	0.03	0.02	0.02
   query9	0.52	0.48	0.48
   query10	0.54	0.53	0.53
   query11	0.12	0.08	0.09
   query12	0.12	0.09	0.09
   query13	0.61	0.62	0.62
   query14	0.78	0.79	0.78
   query15	0.78	0.75	0.75
   query16	0.36	0.39	0.36
   query17	1.03	1.03	1.03
   query18	0.21	0.28	0.21
   query19	1.91	1.84	1.80
   query20	0.01	0.01	0.01
   query21	15.48	0.61	0.55
   query22	2.02	2.28	1.33
   query23	17.08	0.95	0.96
   query24	6.65	1.36	1.04
   query25	0.42	0.09	0.05
   query26	0.75	0.16	0.14
   query27	0.05	0.05	0.03
   query28	6.21	0.84	0.71
   query29	12.63	2.12	2.36
   query30	0.61	0.50	0.52
   query31	2.81	0.39	0.38
   query32	3.40	0.49	0.49
   query33	3.13	3.03	3.09
   query34	15.26	4.78	4.78
   query35	4.87	4.84	4.85
   query36	1.04	1.01	1.01
   query37	0.06	0.05	0.04
   query38	0.04	0.02	0.02
   query39	0.02	0.02	0.01
   query40	0.15	0.14	0.14
   query41	0.07	0.02	0.02
   query42	0.02	0.01	0.01
   query43	0.03	0.02	0.02
   Total cold run time: 103.75 s
   Total hot run time: 30.25 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](statistics)Add synchronize for modify analysisTaskInfoMap and analysisJobInfoMap. #31940 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31967:
URL: https://github.com/apache/doris/pull/31967#issuecomment-1983735633

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit a3e1f441e79b9089f4f553c92dea902e32a3b739 with default session variables
   Stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
   Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
   Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
   Insert into select:       20.9 seconds inserted 10000000 Rows, about 478K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org