You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "suxiaogang223 (via GitHub)" <gi...@apache.org> on 2026/03/11 08:20:14 UTC
[PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
suxiaogang223 opened a new pull request, #61223:
URL: https://github.com/apache/doris/pull/61223
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: N/A
Problem Summary:
When `select ... into outfile` uses `delete_existing_files=true`, parallel outfile writers can race on directory cleanup and delete files uploaded by other writers. This change moves remote outfile cleanup to FE so it happens once before query execution, and clears the delete flag before sending sink options to BE. For `file:///` local outfile, cleanup cannot be coordinated in FE, so the planner falls back to `GATHER` when `delete_existing_files=true` to avoid deletion races. A regression test is added to verify parallel outfile with `delete_existing_files=true` does not lose exported rows.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] Yes. `delete_existing_files=true` is now handled once in FE for remote outfile, and local outfile falls back to serialized execution.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4037556161
<details>
<summary>TPC-H: <b>Total hot run time: 27374 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 175ccf4a693a134c802a3c8be0fffbd50672ee15, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17658 4484 4342 4342
q2 q3 10649 783 547 547
q4 4674 363 259 259
q5 7566 1195 1029 1029
q6 192 179 147 147
q7 767 831 672 672
q8 9353 1475 1304 1304
q9 4912 4738 4642 4642
q10 6330 1908 1599 1599
q11 475 258 244 244
q12 755 569 469 469
q13 18048 2938 2167 2167
q14 231 226 210 210
q15 947 788 785 785
q16 773 747 685 685
q17 704 875 410 410
q18 6210 5412 5188 5188
q19 1113 980 622 622
q20 511 498 391 391
q21 4518 1828 1416 1416
q22 338 284 246 246
Total cold run time: 96724 ms
Total hot run time: 27374 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4421 4338 4352 4338
q2 q3 3837 4296 3750 3750
q4 855 1159 765 765
q5 4051 4261 4347 4261
q6 177 170 140 140
q7 1723 1595 1486 1486
q8 2403 2640 2535 2535
q9 7986 7579 7536 7536
q10 3782 3943 3643 3643
q11 509 432 420 420
q12 489 605 472 472
q13 2723 3144 2294 2294
q14 289 298 280 280
q15 859 810 812 810
q16 752 806 737 737
q17 1209 1465 1324 1324
q18 7092 6760 6649 6649
q19 942 890 948 890
q20 2064 2176 2046 2046
q21 3947 3764 3346 3346
q22 453 460 387 387
Total cold run time: 50563 ms
Total hot run time: 48109 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4043607046
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4107595772
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4037354814
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4064992715
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4101543687
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4106191249
<details>
<summary>TPC-H: <b>Total hot run time: 26636 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ec5b835bd4ababa7b7b84a88941a7bbbbfbe2a89, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17040 4540 4315 4315
q2 q3 10421 795 514 514
q4 4666 366 248 248
q5 7584 1198 1040 1040
q6 174 175 151 151
q7 782 854 665 665
q8 9314 1468 1288 1288
q9 4762 4743 4690 4690
q10 6245 1885 1618 1618
q11 455 254 236 236
q12 700 598 475 475
q13 18029 2907 2187 2187
q14 228 231 207 207
q15 q16 730 731 662 662
q17 744 853 428 428
q18 6298 5257 5254 5254
q19 1137 986 610 610
q20 524 477 368 368
q21 4379 1818 1438 1438
q22 344 300 242 242
Total cold run time: 94556 ms
Total hot run time: 26636 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4940 4674 4668 4668
q2 q3 4014 4393 3834 3834
q4 919 1206 778 778
q5 4057 4377 4362 4362
q6 186 177 145 145
q7 1806 1695 1553 1553
q8 2498 2679 2540 2540
q9 7517 7363 7281 7281
q10 3716 3960 3549 3549
q11 497 466 443 443
q12 515 609 459 459
q13 2803 3207 2360 2360
q14 303 312 269 269
q15 q16 704 750 694 694
q17 1218 1382 1404 1382
q18 7196 6984 6767 6767
q19 908 904 862 862
q20 2044 2180 1963 1963
q21 3977 3485 3336 3336
q22 447 439 374 374
Total cold run time: 50265 ms
Total hot run time: 47619 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4107595867
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4101761353
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/c9026d2ea4686aea2577aeb16d09d0b493ed94c1_c9026d2ea4686aea2577aeb16d09d0b493ed94c1/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/c9026d2ea4686aea2577aeb16d09d0b493ed94c1_c9026d2ea4686aea2577aeb16d09d0b493ed94c1/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.79% (19846/37595) |
| Line Coverage | 36.26% (185172/510624) |
| Region Coverage | 32.55% (143556/440981) |
| Branch Coverage | 33.73% (62814/186210) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4106105852
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4037361468
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4037610215
<details>
<summary>TPC-DS: <b>Total hot run time: 153192 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 175ccf4a693a134c802a3c8be0fffbd50672ee15, data reload: false
query5 4330 662 508 508
query6 334 229 225 225
query7 4219 471 260 260
query8 338 248 233 233
query9 8676 2733 2730 2730
query10 506 360 338 338
query11 7279 5875 5639 5639
query12 179 127 123 123
query13 1257 472 324 324
query14 5671 3904 3560 3560
query14_1 2869 2818 2809 2809
query15 225 213 174 174
query16 987 467 458 458
query17 1121 689 588 588
query18 2424 436 334 334
query19 207 198 175 175
query20 134 127 124 124
query21 227 146 120 120
query22 4839 5090 4778 4778
query23 16567 15960 15860 15860
query23_1 15875 15947 15677 15677
query24 8619 1682 1253 1253
query24_1 1327 1294 1322 1294
query25 536 473 430 430
query26 1239 254 146 146
query27 2799 473 285 285
query28 4528 1859 1846 1846
query29 835 570 476 476
query30 313 246 214 214
query31 1346 1295 1219 1219
query32 79 73 74 73
query33 492 315 281 281
query34 946 930 563 563
query35 631 672 588 588
query36 1027 1145 971 971
query37 140 103 89 89
query38 3004 2943 2859 2859
query39 900 864 853 853
query39_1 878 810 818 810
query40 226 152 136 136
query41 63 61 59 59
query42 301 301 299 299
query43 240 243 224 224
query44
query45 200 194 181 181
query46 885 966 608 608
query47 2130 2145 2053 2053
query48 311 331 242 242
query49 619 453 384 384
query50 692 281 212 212
query51 4207 4106 3989 3989
query52 288 295 279 279
query53 287 338 280 280
query54 288 266 275 266
query55 97 92 82 82
query56 323 322 299 299
query57 1365 1349 1282 1282
query58 293 277 285 277
query59 1334 1419 1250 1250
query60 333 357 324 324
query61 151 145 151 145
query62 632 594 538 538
query63 304 285 277 277
query64 5050 1255 1039 1039
query65
query66 1499 483 368 368
query67 16451 16635 16522 16522
query68
query69 388 315 316 315
query70 1008 976 945 945
query71 347 316 308 308
query72 3008 2915 2595 2595
query73 546 545 329 329
query74 9972 9926 9724 9724
query75 2851 2781 2462 2462
query76 2300 1021 664 664
query77 354 397 308 308
query78 11172 11407 10628 10628
query79 1143 793 586 586
query80 1343 610 515 515
query81 562 281 241 241
query82 1042 149 124 124
query83 326 262 245 245
query84 246 122 99 99
query85 890 465 431 431
query86 423 310 296 296
query87 3172 3106 3044 3044
query88 3571 2655 2640 2640
query89 421 370 361 361
query90 2054 176 172 172
query91 169 162 137 137
query92 82 77 71 71
query93 959 828 518 518
query94 643 316 286 286
query95 598 337 387 337
query96 633 514 233 233
query97 2519 2517 2402 2402
query98 235 223 210 210
query99 994 996 921 921
Total cold run time: 236843 ms
Total hot run time: 153192 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4065838739
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61223_05fd3888073976335cc4c27654038b047dd26527_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61223_05fd3888073976335cc4c27654038b047dd26527_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.15% (26796/36632) |
| Line Coverage | 56.57% (286611/506691) |
| Region Coverage | 53.75% (237950/442716) |
| Branch Coverage | 55.55% (103070/185558) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4043718847
<details>
<summary>TPC-H: <b>Total hot run time: 27566 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ea31080da9f303b191a515636671997010cc3771, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17644 4444 4287 4287
q2 q3 10644 763 511 511
q4 4676 363 247 247
q5 7576 1186 1021 1021
q6 172 174 145 145
q7 775 836 672 672
q8 9297 1446 1302 1302
q9 4922 4645 4663 4645
q10 6297 1891 1641 1641
q11 472 258 241 241
q12 769 559 481 481
q13 18054 2982 2180 2180
q14 232 229 225 225
q15 935 808 807 807
q16 758 718 685 685
q17 713 844 396 396
q18 5858 5419 5229 5229
q19 1118 982 601 601
q20 493 506 391 391
q21 4482 1961 1528 1528
q22 378 333 331 331
Total cold run time: 96265 ms
Total hot run time: 27566 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4792 4624 4562 4562
q2 q3 3909 4443 3805 3805
q4 872 1180 759 759
q5 4052 4444 4345 4345
q6 196 182 165 165
q7 1780 1632 1579 1579
q8 2469 2705 2580 2580
q9 7539 7291 7260 7260
q10 3759 3962 3602 3602
q11 516 463 444 444
q12 501 612 469 469
q13 2916 3117 2325 2325
q14 294 300 269 269
q15 836 789 789 789
q16 710 759 723 723
q17 1121 1340 1321 1321
q18 7224 6868 6761 6761
q19 931 935 857 857
q20 2059 2347 2029 2029
q21 3924 3473 3415 3415
q22 446 425 399 399
Total cold run time: 50846 ms
Total hot run time: 48458 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4106356882
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61223_ec5b835bd4ababa7b7b84a88941a7bbbbfbe2a89_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61223_ec5b835bd4ababa7b7b84a88941a7bbbbfbe2a89_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.26% (26957/36798) |
| Line Coverage | 56.68% (288469/508914) |
| Region Coverage | 53.92% (239963/445032) |
| Branch Coverage | 55.68% (103953/186698) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #61223:
URL: https://github.com/apache/doris/pull/61223#discussion_r2968480600
##########
regression-test/suites/export_p0/test_outfile_parallel_delete_existing_files.groovy:
##########
@@ -0,0 +1,134 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.codehaus.groovy.runtime.IOGroovyMethods
+
+suite("test_outfile_parallel_delete_existing_files", "p0") {
+ StringBuilder strBuilder = new StringBuilder()
+ strBuilder.append("curl --location-trusted -u " + context.config.jdbcUser + ":" + context.config.jdbcPassword)
+ if ((context.config.otherConfigs.get("enableTLS")?.toString()?.equalsIgnoreCase("true")) ?: false) {
+ strBuilder.append(" https://" + context.config.feHttpAddress + "/rest/v1/config/fe")
+ strBuilder.append(" --cert " + context.config.otherConfigs.get("trustCert")
+ + " --cacert " + context.config.otherConfigs.get("trustCACert")
+ + " --key " + context.config.otherConfigs.get("trustCAKey"))
+ } else {
+ strBuilder.append(" http://" + context.config.feHttpAddress + "/rest/v1/config/fe")
+ }
+
+ def process = strBuilder.toString().execute()
+ def code = process.waitFor()
+ def err = IOGroovyMethods.getText(new BufferedReader(new InputStreamReader(process.getErrorStream())))
+ def out = process.getText()
+ logger.info("Request FE Config: code=" + code + ", out=" + out + ", err=" + err)
+ assertEquals(code, 0)
+ def response = parseJson(out.trim())
+ assertEquals(response.code, 0)
+ assertEquals(response.msg, "success")
+ boolean enableDeleteExistingFiles = false
+ for (Object conf : response.data.rows) {
+ assert conf instanceof Map
+ if (((Map<String, String>) conf).get("Name").toLowerCase() == "enable_delete_existing_files") {
+ enableDeleteExistingFiles = ((Map<String, String>) conf).get("Value").toLowerCase() == "true"
+ }
+ }
+ if (!enableDeleteExistingFiles) {
+ logger.warn("Please set enable_delete_existing_files to true to run test_outfile_parallel_delete_existing_files")
+ return
+ }
+
+ String ak = getS3AK()
+ String sk = getS3SK()
+ String s3Endpoint = getS3Endpoint()
+ String region = getS3Region()
+ String bucket = context.config.otherConfigs.get("s3BucketName")
+ String provider = getS3Provider()
+ String tableName = "test_outfile_parallel_delete_existing_files"
+ String uuid = UUID.randomUUID().toString()
+ String outFilePath = "${bucket}/outfile/parallel_delete_existing_files/${uuid}/exp_"
+
+ def exportToS3 = { String filterSql, boolean deleteExistingFiles ->
+ String deleteProperty = deleteExistingFiles ? "\"delete_existing_files\" = \"true\"," : ""
+ sql """
+ SELECT * FROM ${tableName} ${filterSql}
+ INTO OUTFILE "s3://${outFilePath}"
+ FORMAT AS csv
+ PROPERTIES (
+ ${deleteProperty}
+ "column_separator" = ",",
+ "s3.endpoint" = "${s3Endpoint}",
+ "s3.region" = "${region}",
+ "s3.secret_key" = "${sk}",
+ "s3.access_key" = "${ak}",
+ "provider" = "${provider}"
+ );
+ """
+ }
+
+ try {
+ sql """ set enable_nereids_planner = true """
+ sql """ set enable_fallback_to_original_planner = false """
+ sql """ set enable_parallel_outfile = true """
+ sql """ set parallel_pipeline_task_num = 8 """
+
+ sql """ DROP TABLE IF EXISTS ${tableName} """
+ sql """
+ CREATE TABLE ${tableName} (
+ id INT NOT NULL,
+ name STRING NOT NULL,
+ score INT NOT NULL
+ )
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 16
+ PROPERTIES("replication_num" = "1");
+ """
+
+ sql """
+ INSERT INTO ${tableName}
+ SELECT
+ number AS id,
+ concat('name_', cast(number AS string)) AS name,
+ cast(number % 97 AS int) AS score
+ FROM numbers("number" = "20000");
+ """
+
+ def expected = sql """ SELECT count(*), sum(id), sum(score) FROM ${tableName}; """
+
+ exportToS3("WHERE id < 5000", false)
+ exportToS3("", true)
+
+ def actual = sql """
+ SELECT count(*), sum(id), sum(score) FROM S3(
+ "uri" = "s3://${outFilePath}*",
+ "s3.endpoint" = "${s3Endpoint}",
+ "s3.region" = "${region}",
+ "s3.secret_key" = "${sk}",
+ "s3.access_key" = "${ak}",
+ "provider" = "${provider}",
+ "format" = "csv",
+ "column_separator" = ",",
+ "csv_schema" = "id:int;name:string;score:int"
+ );
+ """
+
+ assertEquals(expected[0][0], actual[0][0])
+ assertEquals(expected[0][1], actual[0][1])
+ assertEquals(expected[0][2], actual[0][2])
+ } finally {
+ try_sql(""" set enable_parallel_outfile = false """)
+ try_sql(""" DROP TABLE IF EXISTS ${tableName} """)
Review Comment:
Minor: Per regression test standards, tables should be dropped *before* use (not after) to preserve the debugging environment. The `DROP TABLE IF EXISTS` before `CREATE` on line 87 is correct, but this `finally` block cleanup should be removed.
##########
regression-test/suites/export_p0/test_outfile_parallel_delete_existing_files.groovy:
##########
@@ -0,0 +1,134 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.codehaus.groovy.runtime.IOGroovyMethods
+
+suite("test_outfile_parallel_delete_existing_files", "p0") {
+ StringBuilder strBuilder = new StringBuilder()
+ strBuilder.append("curl --location-trusted -u " + context.config.jdbcUser + ":" + context.config.jdbcPassword)
+ if ((context.config.otherConfigs.get("enableTLS")?.toString()?.equalsIgnoreCase("true")) ?: false) {
+ strBuilder.append(" https://" + context.config.feHttpAddress + "/rest/v1/config/fe")
+ strBuilder.append(" --cert " + context.config.otherConfigs.get("trustCert")
+ + " --cacert " + context.config.otherConfigs.get("trustCACert")
+ + " --key " + context.config.otherConfigs.get("trustCAKey"))
+ } else {
+ strBuilder.append(" http://" + context.config.feHttpAddress + "/rest/v1/config/fe")
+ }
+
+ def process = strBuilder.toString().execute()
+ def code = process.waitFor()
+ def err = IOGroovyMethods.getText(new BufferedReader(new InputStreamReader(process.getErrorStream())))
+ def out = process.getText()
+ logger.info("Request FE Config: code=" + code + ", out=" + out + ", err=" + err)
+ assertEquals(code, 0)
+ def response = parseJson(out.trim())
+ assertEquals(response.code, 0)
+ assertEquals(response.msg, "success")
+ boolean enableDeleteExistingFiles = false
+ for (Object conf : response.data.rows) {
+ assert conf instanceof Map
+ if (((Map<String, String>) conf).get("Name").toLowerCase() == "enable_delete_existing_files") {
+ enableDeleteExistingFiles = ((Map<String, String>) conf).get("Value").toLowerCase() == "true"
+ }
+ }
+ if (!enableDeleteExistingFiles) {
+ logger.warn("Please set enable_delete_existing_files to true to run test_outfile_parallel_delete_existing_files")
+ return
+ }
+
+ String ak = getS3AK()
+ String sk = getS3SK()
+ String s3Endpoint = getS3Endpoint()
+ String region = getS3Region()
+ String bucket = context.config.otherConfigs.get("s3BucketName")
+ String provider = getS3Provider()
+ String tableName = "test_outfile_parallel_delete_existing_files"
+ String uuid = UUID.randomUUID().toString()
+ String outFilePath = "${bucket}/outfile/parallel_delete_existing_files/${uuid}/exp_"
+
+ def exportToS3 = { String filterSql, boolean deleteExistingFiles ->
+ String deleteProperty = deleteExistingFiles ? "\"delete_existing_files\" = \"true\"," : ""
+ sql """
+ SELECT * FROM ${tableName} ${filterSql}
+ INTO OUTFILE "s3://${outFilePath}"
+ FORMAT AS csv
+ PROPERTIES (
+ ${deleteProperty}
+ "column_separator" = ",",
+ "s3.endpoint" = "${s3Endpoint}",
+ "s3.region" = "${region}",
+ "s3.secret_key" = "${sk}",
+ "s3.access_key" = "${ak}",
+ "provider" = "${provider}"
+ );
+ """
+ }
+
+ try {
+ sql """ set enable_nereids_planner = true """
Review Comment:
Minor: Per regression test standards, "other variables do not need extra setup before testing. For example, nereids optimizer and pipeline engine settings can use default states." These two lines (`set enable_nereids_planner = true`, `set enable_fallback_to_original_planner = false`) are unnecessary since Nereids is the default planner.
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java:
##########
@@ -184,6 +184,10 @@ public long getMaxFileSizeBytes() {
return maxFileSizeBytes;
}
+ public boolean shouldDeleteExistingFiles() {
Review Comment:
**CRITICAL: Unresolved merge conflict markers.** Lines 776-779 contain `<<<<<<<`, `=======`, `>>>>>>>` markers from a failed merge/rebase. This will cause a compilation failure. These must be removed before merge.
The fix is simple — just remove these four lines (the class closing brace at line 775 is the correct end of file).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #61223:
URL: https://github.com/apache/doris/pull/61223
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4065239095
<details>
<summary>TPC-H: <b>Total hot run time: 27022 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 05fd3888073976335cc4c27654038b047dd26527, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17600 4465 4346 4346
q2 q3 10732 810 560 560
q4 4721 369 255 255
q5 8091 1204 1006 1006
q6 227 184 145 145
q7 821 857 668 668
q8 10726 1505 1315 1315
q9 6551 4782 4743 4743
q10 6472 1928 1676 1676
q11 457 245 232 232
q12 767 582 468 468
q13 18066 2968 2186 2186
q14 234 234 218 218
q15 q16 748 747 674 674
q17 751 837 476 476
q18 6006 5439 5236 5236
q19 1131 977 635 635
q20 541 494 384 384
q21 4889 2074 1531 1531
q22 405 343 268 268
Total cold run time: 99936 ms
Total hot run time: 27022 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4718 4674 4698 4674
q2 q3 3955 4342 3836 3836
q4 883 1229 826 826
q5 4124 4454 4397 4397
q6 179 179 141 141
q7 1759 1645 1572 1572
q8 2507 2702 2582 2582
q9 7530 7386 7482 7386
q10 3868 4180 3595 3595
q11 515 431 422 422
q12 504 635 523 523
q13 2731 3172 2425 2425
q14 328 299 264 264
q15 q16 736 748 731 731
q17 1192 1371 1333 1333
q18 7343 6864 6856 6856
q19 939 883 919 883
q20 2085 2176 2009 2009
q21 4051 3538 3408 3408
q22 420 424 376 376
Total cold run time: 50367 ms
Total hot run time: 48239 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4065275757
<details>
<summary>TPC-DS: <b>Total hot run time: 168654 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 05fd3888073976335cc4c27654038b047dd26527, data reload: false
query5 4321 633 498 498
query6 327 231 202 202
query7 4212 462 269 269
query8 332 238 229 229
query9 8712 2747 2751 2747
query10 522 377 332 332
query11 6962 5161 4946 4946
query12 185 132 129 129
query13 1274 468 358 358
query14 5608 3747 3492 3492
query14_1 2860 2807 2829 2807
query15 211 197 177 177
query16 971 470 427 427
query17 888 723 619 619
query18 2443 456 348 348
query19 212 214 188 188
query20 138 137 128 128
query21 214 132 112 112
query22 13312 13994 14575 13994
query23 16320 15817 15822 15817
query23_1 15773 15663 15619 15619
query24 7338 1607 1213 1213
query24_1 1219 1228 1242 1228
query25 532 460 445 445
query26 1242 262 144 144
query27 2789 476 306 306
query28 4457 1879 1854 1854
query29 838 552 469 469
query30 296 231 196 196
query31 1012 945 862 862
query32 84 70 71 70
query33 507 335 278 278
query34 898 863 546 546
query35 644 687 596 596
query36 1061 1131 997 997
query37 136 95 78 78
query38 2905 2962 2853 2853
query39 871 850 833 833
query39_1 792 802 795 795
query40 232 152 137 137
query41 64 58 59 58
query42 256 256 265 256
query43 238 250 222 222
query44
query45 196 186 179 179
query46 881 982 606 606
query47 2103 2112 2037 2037
query48 312 313 230 230
query49 630 454 378 378
query50 687 276 224 224
query51 4077 4049 4033 4033
query52 267 264 256 256
query53 299 345 286 286
query54 298 268 272 268
query55 90 89 86 86
query56 324 315 322 315
query57 1939 1809 1548 1548
query58 282 271 265 265
query59 2819 2980 2748 2748
query60 332 339 324 324
query61 155 149 148 148
query62 602 601 524 524
query63 307 277 277 277
query64 5107 1278 978 978
query65
query66 1472 480 367 367
query67 24395 24368 24435 24368
query68
query69 403 311 295 295
query70 999 942 971 942
query71 332 308 290 290
query72 2892 2769 2347 2347
query73 556 545 325 325
query74 9642 9566 9433 9433
query75 2854 2742 2496 2496
query76 2296 1051 682 682
query77 358 394 308 308
query78 10974 11130 10475 10475
query79 1190 829 576 576
query80 1284 616 543 543
query81 543 262 234 234
query82 982 151 115 115
query83 340 260 238 238
query84 296 119 101 101
query85 893 477 426 426
query86 399 303 300 300
query87 3141 3102 3019 3019
query88 3590 2669 2656 2656
query89 420 371 344 344
query90 2001 180 176 176
query91 171 167 135 135
query92 75 76 67 67
query93 1010 861 504 504
query94 643 298 289 289
query95 584 338 315 315
query96 636 530 235 235
query97 2466 2491 2438 2438
query98 243 227 224 224
query99 1025 985 910 910
Total cold run time: 249240 ms
Total hot run time: 168654 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4064312954
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4106239147
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/ec5b835bd4ababa7b7b84a88941a7bbbbfbe2a89_ec5b835bd4ababa7b7b84a88941a7bbbbfbe2a89/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/ec5b835bd4ababa7b7b84a88941a7bbbbfbe2a89_ec5b835bd4ababa7b7b84a88941a7bbbbfbe2a89/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.79% (19846/37595) |
| Line Coverage | 36.27% (185183/510624) |
| Region Coverage | 32.55% (143525/440981) |
| Branch Coverage | 33.73% (62808/186210) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4037806766
# FE UT Coverage Report
Increment line coverage ` 0.00% (0/24)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61223_175ccf4a693a134c802a3c8be0fffbd50672ee15/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61223_175ccf4a693a134c802a3c8be0fffbd50672ee15/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4101542913
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #61223:
URL: https://github.com/apache/doris/pull/61223#discussion_r2937652231
##########
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##########
@@ -1544,6 +1548,19 @@ private void outfileWriteSuccess(OutFileClause outFileClause) throws Exception {
}
}
+ private void deleteExistingOutfileFilesInFe(OutFileClause outFileClause) throws UserException {
+ // Handle directory cleanup once in FE so parallel outfile writers never race on deletion.
+ if (!outFileClause.shouldDeleteExistingFiles()) {
+ return;
+ }
+ if (outFileClause.getBrokerDesc() == null
+ || outFileClause.getBrokerDesc().storageType() == StorageType.LOCAL) {
+ return;
+ }
+ BrokerUtil.deleteParentDirectoryWithFileSystem(outFileClause.getFilePath(), outFileClause.getBrokerDesc());
+ outFileClause.markDeleteExistingFilesHandledInFe();
+ }
Review Comment:
**[Dead Code]** `markDeleteExistingFilesHandledInFe()` sets `deleteExistingFiles = false` on this ephemeral `OutFileClause` instance, but:
1. This instance is never used again (it was created fresh by `getOutFileClause()` and passed as a parameter).
2. The `TResultFileSinkOptions` that carries `deleteExistingFiles` to BE was already created in `PhysicalPlanTranslator.visitPhysicalFileSink()` during plan translation, from a different `OutFileClause` instance.
This call has no observable effect.
##########
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##########
@@ -1368,6 +1369,9 @@ public void executeAndSendResult(boolean isOutfileQuery, boolean isSendFields,
coordBase.setIsProfileSafeStmt(this.isProfileSafeStmt());
try {
+ if (isOutfileQuery) {
+ deleteExistingOutfileFilesInFe(queryStmt.getOutFileClause());
+ }
Review Comment:
**[Critical Bug]** `queryStmt.getOutFileClause()` returns a **new `OutFileClause` object on every call** (see `LogicalPlanAdapter.getOutFileClause()` which constructs `new OutFileClause(...)` each time). This means `markDeleteExistingFilesHandledInFe()` inside `deleteExistingOutfileFilesInFe()` mutates a throwaway object — the flag is never cleared on the `OutFileClause` (or more critically, the `TResultFileSinkOptions`) that is actually sent to BE.
Moreover, the `ResultFileSink` containing `TResultFileSinkOptions` with `deleteExistingFiles=true` is already baked into the plan fragments during `planner.plan()` (in `PhysicalPlanTranslator.visitPhysicalFileSink()`), which runs well before this line. Even reaching the "right" `OutFileClause` wouldn't help — the Thrift options struct was already copied.
**Result**: FE deletes the directory, but each parallel BE writer still receives `deleteExistingFiles=true` and will race on deletion — the original bug is NOT fixed.
**Suggested fix**: Either (a) clear `deleteExistingFiles` on the `ResultFileSink.fileSinkOptions` in the plan fragment after the FE deletion, or (b) move the deletion + flag clearing to **before** `planner.plan()` by modifying `LogicalFileSink.properties` to remove the `delete_existing_files` key after performing the FE-side deletion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4065104401
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/05fd3888073976335cc4c27654038b047dd26527_05fd3888073976335cc4c27654038b047dd26527_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/05fd3888073976335cc4c27654038b047dd26527_05fd3888073976335cc4c27654038b047dd26527_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 79.24% (1798/2269) |
| Line Coverage | 64.52% (32275/50026) |
| Region Coverage | 65.43% (16163/24702) |
| Branch Coverage | 55.81% (8604/15416) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4106205766
<details>
<summary>TPC-DS: <b>Total hot run time: 168180 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ec5b835bd4ababa7b7b84a88941a7bbbbfbe2a89, data reload: false
query5 4326 631 490 490
query6 350 242 224 224
query7 4240 458 267 267
query8 339 251 229 229
query9 8686 2676 2683 2676
query10 534 381 352 352
query11 6938 5086 4856 4856
query12 181 126 124 124
query13 1271 456 350 350
query14 5763 3673 3421 3421
query14_1 2793 2809 2788 2788
query15 205 193 177 177
query16 983 452 436 436
query17 1119 730 633 633
query18 2459 451 349 349
query19 218 211 187 187
query20 141 130 126 126
query21 216 136 116 116
query22 13308 14125 14888 14125
query23 16177 15924 15798 15798
query23_1 15591 15587 15539 15539
query24 7016 1601 1224 1224
query24_1 1219 1227 1229 1227
query25 540 451 407 407
query26 1243 268 144 144
query27 2775 477 293 293
query28 4394 1803 1795 1795
query29 830 561 508 508
query30 304 223 192 192
query31 1004 943 849 849
query32 82 72 72 72
query33 514 330 276 276
query34 868 861 516 516
query35 649 693 618 618
query36 1104 1150 941 941
query37 128 94 86 86
query38 2935 2916 2863 2863
query39 854 846 799 799
query39_1 791 795 795 795
query40 234 154 135 135
query41 62 58 58 58
query42 258 254 260 254
query43 235 243 228 228
query44
query45 201 187 180 180
query46 865 981 610 610
query47 2145 2131 2107 2107
query48 313 311 222 222
query49 651 450 380 380
query50 690 277 221 221
query51 4058 4049 4008 4008
query52 259 263 256 256
query53 292 327 285 285
query54 302 267 268 267
query55 94 90 84 84
query56 312 320 315 315
query57 1915 1833 1548 1548
query58 285 283 264 264
query59 2765 2955 2762 2762
query60 346 337 315 315
query61 156 148 152 148
query62 628 582 527 527
query63 305 277 274 274
query64 4991 1284 1003 1003
query65
query66 1447 449 355 355
query67 24119 24162 24129 24129
query68
query69 393 305 291 291
query70 976 961 952 952
query71 332 306 297 297
query72 2863 2682 2619 2619
query73 548 553 314 314
query74 9611 9551 9401 9401
query75 2866 2755 2468 2468
query76 2207 1033 679 679
query77 374 386 317 317
query78 10829 10988 10435 10435
query79 2708 778 569 569
query80 1814 677 551 551
query81 552 267 220 220
query82 946 151 120 120
query83 325 261 245 245
query84 251 115 100 100
query85 908 490 453 453
query86 432 310 293 293
query87 3147 3082 3014 3014
query88 3518 2632 2626 2626
query89 428 383 342 342
query90 1990 179 170 170
query91 173 171 134 134
query92 75 78 68 68
query93 1275 827 493 493
query94 637 281 284 281
query95 580 337 311 311
query96 646 523 231 231
query97 2479 2482 2410 2410
query98 238 223 218 218
query99 1037 994 924 924
Total cold run time: 250639 ms
Total hot run time: 168180 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4101659934
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/c9026d2ea4686aea2577aeb16d09d0b493ed94c1_c9026d2ea4686aea2577aeb16d09d0b493ed94c1_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/c9026d2ea4686aea2577aeb16d09d0b493ed94c1_c9026d2ea4686aea2577aeb16d09d0b493ed94c1_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.63% (1796/2284) |
| Line Coverage | 64.39% (32281/50130) |
| Region Coverage | 65.30% (16169/24760) |
| Branch Coverage | 55.71% (8611/15456) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4043749246
<details>
<summary>TPC-DS: <b>Total hot run time: 153156 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ea31080da9f303b191a515636671997010cc3771, data reload: false
query5 4352 658 525 525
query6 317 226 213 213
query7 4222 459 263 263
query8 352 248 238 238
query9 8739 2727 2722 2722
query10 481 408 357 357
query11 7374 5867 5558 5558
query12 195 127 125 125
query13 1273 465 350 350
query14 5773 3826 3584 3584
query14_1 2790 2810 2786 2786
query15 203 193 182 182
query16 986 477 376 376
query17 1120 713 626 626
query18 2444 445 353 353
query19 209 208 189 189
query20 132 131 126 126
query21 224 144 126 126
query22 4931 4988 4926 4926
query23 16922 16122 15896 15896
query23_1 15886 15819 15849 15819
query24 8657 1690 1311 1311
query24_1 1308 1317 1292 1292
query25 626 514 454 454
query26 1756 285 162 162
query27 2790 466 285 285
query28 4480 1832 1833 1832
query29 860 556 471 471
query30 312 246 211 211
query31 1356 1296 1205 1205
query32 82 72 70 70
query33 502 331 270 270
query34 931 915 574 574
query35 640 681 599 599
query36 1099 1094 940 940
query37 123 91 83 83
query38 2960 2937 2848 2848
query39 899 994 856 856
query39_1 847 816 844 816
query40 235 152 134 134
query41 77 69 69 69
query42 309 288 296 288
query43 232 238 220 220
query44
query45 203 194 180 180
query46 885 976 614 614
query47 2173 2136 2053 2053
query48 302 310 232 232
query49 640 471 371 371
query50 687 279 210 210
query51 4111 4090 4011 4011
query52 291 292 286 286
query53 287 331 279 279
query54 289 266 264 264
query55 91 91 81 81
query56 303 327 309 309
query57 1367 1352 1292 1292
query58 304 280 271 271
query59 1318 1459 1296 1296
query60 366 337 310 310
query61 152 146 146 146
query62 624 588 532 532
query63 312 280 274 274
query64 5146 1242 987 987
query65
query66 1457 463 356 356
query67 16345 16390 16320 16320
query68
query69 414 304 277 277
query70 1010 854 970 854
query71 337 317 291 291
query72 2850 2633 2471 2471
query73 534 545 325 325
query74 10012 9907 9791 9791
query75 2876 2741 2465 2465
query76 2314 1011 665 665
query77 361 385 318 318
query78 11192 11329 10643 10643
query79 2604 773 598 598
query80 1784 632 565 565
query81 572 281 247 247
query82 984 155 117 117
query83 342 266 244 244
query84 245 111 97 97
query85 899 475 425 425
query86 413 305 328 305
query87 3119 3108 3030 3030
query88 3524 2636 2610 2610
query89 425 361 346 346
query90 2021 180 172 172
query91 164 157 130 130
query92 73 78 72 72
query93 1048 832 500 500
query94 634 317 302 302
query95 575 330 320 320
query96 634 511 230 230
query97 2517 2474 2457 2457
query98 238 230 224 224
query99 969 997 907 907
Total cold run time: 238444 ms
Total hot run time: 153156 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4064900000
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](outfile) handle delete_existing_files before parallel export [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61223:
URL: https://github.com/apache/doris/pull/61223#issuecomment-4065806590
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/05fd3888073976335cc4c27654038b047dd26527_05fd3888073976335cc4c27654038b047dd26527/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/05fd3888073976335cc4c27654038b047dd26527_05fd3888073976335cc4c27654038b047dd26527/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.65% (19703/37426) |
| Line Coverage | 36.20% (184019/508404) |
| Region Coverage | 32.36% (141960/438667) |
| Branch Coverage | 33.56% (62108/185076) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org