You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "kaka11chen (via GitHub)" <gi...@apache.org> on 2024/03/28 07:09:10 UTC

[PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]

kaka11chen opened a new pull request, #32959:
URL: https://github.com/apache/doris/pull/32959

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024611302

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 181738 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit dd90aaa6f2e0d7c59e0e8a7cc472144cc94b7aad, data reload: false
   
   query1	951	375	342	342
   query2	6540	1946	1928	1928
   query3	6702	212	210	210
   query4	31816	21263	21206	21206
   query5	4283	397	392	392
   query6	265	177	169	169
   query7	4629	294	290	290
   query8	232	169	168	168
   query9	9282	2359	2315	2315
   query10	554	241	255	241
   query11	17238	14205	14297	14205
   query12	143	91	91	91
   query13	1629	410	422	410
   query14	9948	7976	7975	7975
   query15	262	199	174	174
   query16	8259	258	257	257
   query17	2019	592	532	532
   query18	2087	283	269	269
   query19	339	161	155	155
   query20	92	90	84	84
   query21	196	128	117	117
   query22	5136	4858	4832	4832
   query23	33790	32812	32748	32748
   query24	10772	2904	2880	2880
   query25	597	357	372	357
   query26	1171	150	152	150
   query27	2833	348	354	348
   query28	7408	1864	1865	1864
   query29	870	620	620	620
   query30	300	146	147	146
   query31	997	730	731	730
   query32	93	58	57	57
   query33	775	248	248	248
   query34	1040	490	485	485
   query35	819	603	603	603
   query36	1049	895	872	872
   query37	131	62	61	61
   query38	3585	3409	3450	3409
   query39	1473	1444	1412	1412
   query40	211	114	109	109
   query41	47	44	44	44
   query42	105	94	97	94
   query43	479	471	473	471
   query44	1213	718	726	718
   query45	284	272	265	265
   query46	1086	683	673	673
   query47	1951	1884	1828	1828
   query48	431	351	346	346
   query49	1088	329	324	324
   query50	759	366	359	359
   query51	6878	6727	6860	6727
   query52	105	92	97	92
   query53	336	277	271	271
   query54	282	232	222	222
   query55	84	78	75	75
   query56	237	219	221	219
   query57	1228	1158	1148	1148
   query58	231	211	211	211
   query59	2906	2731	2566	2566
   query60	259	230	239	230
   query61	99	93	94	93
   query62	640	429	408	408
   query63	300	277	275	275
   query64	5724	4185	4039	4039
   query65	3068	3018	3017	3017
   query66	876	389	368	368
   query67	15388	14993	14780	14780
   query68	5223	506	512	506
   query69	545	366	368	366
   query70	1280	1156	1170	1156
   query71	419	274	267	267
   query72	6370	2678	2504	2504
   query73	713	320	319	319
   query74	7559	6429	6393	6393
   query75	3019	2201	2216	2201
   query76	3512	896	948	896
   query77	402	265	269	265
   query78	10949	10280	10183	10183
   query79	8269	534	535	534
   query80	2271	390	408	390
   query81	560	226	221	221
   query82	1552	91	101	91
   query83	277	156	142	142
   query84	287	85	87	85
   query85	2120	331	311	311
   query86	496	311	283	283
   query87	3750	3543	3566	3543
   query88	4921	2313	2299	2299
   query89	483	370	371	370
   query90	2002	176	176	176
   query91	171	133	140	133
   query92	62	49	48	48
   query93	5923	494	484	484
   query94	1239	175	178	175
   query95	437	336	330	330
   query96	605	273	265	265
   query97	2669	2500	2469	2469
   query98	235	219	209	209
   query99	1135	800	830	800
   Total cold run time: 306055 ms
   Total hot run time: 181738 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024914229

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]

Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #32959:
URL: https://github.com/apache/doris/pull/32959


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024625077

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit dd90aaa6f2e0d7c59e0e8a7cc472144cc94b7aad with default session variables
   Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
   Insert into select:       13.6 seconds inserted 10000000 Rows, about 735K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024546911

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
   See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024914270

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024588758

   TeamCity be ut coverage result:
    Function Coverage: 35.63% (8835/24797) 
    Line Coverage: 27.34% (72395/264815)
    Region Coverage: 26.56% (37542/141351)
    Branch Coverage: 23.37% (19149/81948)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/dd90aaa6f2e0d7c59e0e8a7cc472144cc94b7aad_dd90aaa6f2e0d7c59e0e8a7cc472144cc94b7aad/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]

Posted by "kaka11chen (via GitHub)" <gi...@apache.org>.
kaka11chen commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024546989

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024553427

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024593770

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 37262 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit dd90aaa6f2e0d7c59e0e8a7cc472144cc94b7aad, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17607	4187	4062	4062
   q2	2116	158	158	158
   q3	10583	1087	1163	1087
   q4	10232	794	730	730
   q5	7446	2890	2870	2870
   q6	197	120	122	120
   q7	1025	577	553	553
   q8	9338	1958	1935	1935
   q9	6918	6315	6310	6310
   q10	8428	3500	3565	3500
   q11	447	215	215	215
   q12	410	197	190	190
   q13	17784	2859	2858	2858
   q14	222	210	213	210
   q15	508	459	468	459
   q16	487	369	362	362
   q17	934	589	532	532
   q18	7112	6396	6446	6396
   q19	2469	1417	1440	1417
   q20	545	254	249	249
   q21	3493	2888	2763	2763
   q22	345	288	286	286
   Total cold run time: 108646 ms
   Total hot run time: 37262 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4104	4044	4039	4039
   q2	329	229	242	229
   q3	2957	2769	2803	2769
   q4	1812	1547	1542	1542
   q5	5174	5201	5212	5201
   q6	195	121	119	119
   q7	2172	1810	1804	1804
   q8	3142	3292	3271	3271
   q9	8381	8345	8356	8345
   q10	3802	3757	3792	3757
   q11	541	439	446	439
   q12	692	566	610	566
   q13	16909	2834	2821	2821
   q14	276	250	253	250
   q15	486	458	453	453
   q16	462	420	425	420
   q17	1716	1471	1427	1427
   q18	7417	7143	7159	7143
   q19	1605	1508	1546	1508
   q20	1891	1724	1708	1708
   q21	4803	4566	4593	4566
   q22	518	432	445	432
   Total cold run time: 69384 ms
   Total hot run time: 52809 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org