You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "kaka11chen (via GitHub)" <gi...@apache.org> on 2024/03/28 07:09:10 UTC
[PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]
kaka11chen opened a new pull request, #32959:
URL: https://github.com/apache/doris/pull/32959
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024611302
<details>
<summary>TPC-DS: <b>Total hot run time: 181738 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dd90aaa6f2e0d7c59e0e8a7cc472144cc94b7aad, data reload: false
query1 951 375 342 342
query2 6540 1946 1928 1928
query3 6702 212 210 210
query4 31816 21263 21206 21206
query5 4283 397 392 392
query6 265 177 169 169
query7 4629 294 290 290
query8 232 169 168 168
query9 9282 2359 2315 2315
query10 554 241 255 241
query11 17238 14205 14297 14205
query12 143 91 91 91
query13 1629 410 422 410
query14 9948 7976 7975 7975
query15 262 199 174 174
query16 8259 258 257 257
query17 2019 592 532 532
query18 2087 283 269 269
query19 339 161 155 155
query20 92 90 84 84
query21 196 128 117 117
query22 5136 4858 4832 4832
query23 33790 32812 32748 32748
query24 10772 2904 2880 2880
query25 597 357 372 357
query26 1171 150 152 150
query27 2833 348 354 348
query28 7408 1864 1865 1864
query29 870 620 620 620
query30 300 146 147 146
query31 997 730 731 730
query32 93 58 57 57
query33 775 248 248 248
query34 1040 490 485 485
query35 819 603 603 603
query36 1049 895 872 872
query37 131 62 61 61
query38 3585 3409 3450 3409
query39 1473 1444 1412 1412
query40 211 114 109 109
query41 47 44 44 44
query42 105 94 97 94
query43 479 471 473 471
query44 1213 718 726 718
query45 284 272 265 265
query46 1086 683 673 673
query47 1951 1884 1828 1828
query48 431 351 346 346
query49 1088 329 324 324
query50 759 366 359 359
query51 6878 6727 6860 6727
query52 105 92 97 92
query53 336 277 271 271
query54 282 232 222 222
query55 84 78 75 75
query56 237 219 221 219
query57 1228 1158 1148 1148
query58 231 211 211 211
query59 2906 2731 2566 2566
query60 259 230 239 230
query61 99 93 94 93
query62 640 429 408 408
query63 300 277 275 275
query64 5724 4185 4039 4039
query65 3068 3018 3017 3017
query66 876 389 368 368
query67 15388 14993 14780 14780
query68 5223 506 512 506
query69 545 366 368 366
query70 1280 1156 1170 1156
query71 419 274 267 267
query72 6370 2678 2504 2504
query73 713 320 319 319
query74 7559 6429 6393 6393
query75 3019 2201 2216 2201
query76 3512 896 948 896
query77 402 265 269 265
query78 10949 10280 10183 10183
query79 8269 534 535 534
query80 2271 390 408 390
query81 560 226 221 221
query82 1552 91 101 91
query83 277 156 142 142
query84 287 85 87 85
query85 2120 331 311 311
query86 496 311 283 283
query87 3750 3543 3566 3543
query88 4921 2313 2299 2299
query89 483 370 371 370
query90 2002 176 176 176
query91 171 133 140 133
query92 62 49 48 48
query93 5923 494 484 484
query94 1239 175 178 175
query95 437 336 330 330
query96 605 273 265 265
query97 2669 2500 2469 2469
query98 235 219 209 209
query99 1135 800 830 800
Total cold run time: 306055 ms
Total hot run time: 181738 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024914229
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #32959:
URL: https://github.com/apache/doris/pull/32959
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024625077
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit dd90aaa6f2e0d7c59e0e8a7cc472144cc94b7aad with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select: 13.6 seconds inserted 10000000 Rows, about 735K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024546911
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024914270
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024588758
TeamCity be ut coverage result:
Function Coverage: 35.63% (8835/24797)
Line Coverage: 27.34% (72395/264815)
Region Coverage: 26.56% (37542/141351)
Branch Coverage: 23.37% (19149/81948)
Coverage Report: http://coverage.selectdb-in.cc/coverage/dd90aaa6f2e0d7c59e0e8a7cc472144cc94b7aad_dd90aaa6f2e0d7c59e0e8a7cc472144cc94b7aad/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]
Posted by "kaka11chen (via GitHub)" <gi...@apache.org>.
kaka11chen commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024546989
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024553427
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](hive-writer) Fix correct num when hive writing data to an unpartitioned table if size large than `hive_sink_max_file_size`. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32959:
URL: https://github.com/apache/doris/pull/32959#issuecomment-2024593770
<details>
<summary>TPC-H: <b>Total hot run time: 37262 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dd90aaa6f2e0d7c59e0e8a7cc472144cc94b7aad, data reload: false
------ Round 1 ----------------------------------
q1 17607 4187 4062 4062
q2 2116 158 158 158
q3 10583 1087 1163 1087
q4 10232 794 730 730
q5 7446 2890 2870 2870
q6 197 120 122 120
q7 1025 577 553 553
q8 9338 1958 1935 1935
q9 6918 6315 6310 6310
q10 8428 3500 3565 3500
q11 447 215 215 215
q12 410 197 190 190
q13 17784 2859 2858 2858
q14 222 210 213 210
q15 508 459 468 459
q16 487 369 362 362
q17 934 589 532 532
q18 7112 6396 6446 6396
q19 2469 1417 1440 1417
q20 545 254 249 249
q21 3493 2888 2763 2763
q22 345 288 286 286
Total cold run time: 108646 ms
Total hot run time: 37262 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4104 4044 4039 4039
q2 329 229 242 229
q3 2957 2769 2803 2769
q4 1812 1547 1542 1542
q5 5174 5201 5212 5201
q6 195 121 119 119
q7 2172 1810 1804 1804
q8 3142 3292 3271 3271
q9 8381 8345 8356 8345
q10 3802 3757 3792 3757
q11 541 439 446 439
q12 692 566 610 566
q13 16909 2834 2821 2821
q14 276 250 253 250
q15 486 458 453 453
q16 462 420 425 420
q17 1716 1471 1427 1427
q18 7417 7143 7159 7143
q19 1605 1508 1546 1508
q20 1891 1724 1708 1708
q21 4803 4566 4593 4566
q22 518 432 445 432
Total cold run time: 69384 ms
Total hot run time: 52809 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org