You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morrySnow (via GitHub)" <gi...@apache.org> on 2026/02/28 09:38:55 UTC
[PR] [fix](insert-overwrite) skip deleted partitions before replace to avoid DdlException [doris]
morrySnow opened a new pull request, #60914:
URL: https://github.com/apache/doris/pull/60914
### What problem does this PR solve?
Problem Summary:
When performing insert overwrite on a whole table, partition names are snapshotted at planning time without holding the table lock. A concurrent DROP PARTITION between the snapshot and the replace step causes Env.replaceTempPartition() to throw DdlException because the original partition no longer exists.
Fix: after acquiring the table write lock in InsertOverwriteUtil.replacePartition(), filter out any original partitions that have been deleted before constructing ReplacePartitionOp. The temp partition list is kept intact so all written data remains visible.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert-overwrite) skip deleted partitions before replace to avoid DdlException [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #60914:
URL: https://github.com/apache/doris/pull/60914#issuecomment-3976748620
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert-overwrite) skip deleted partitions before replace to avoid DdlException [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60914:
URL: https://github.com/apache/doris/pull/60914#issuecomment-3976774892
<details>
<summary>TPC-H: <b>Total hot run time: 28900 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 160ee55af10a7aaff412ee4c81e0034a9cca2dd2, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17603 4465 4309 4309
q2 q3 10654 800 524 524
q4 4677 356 267 267
q5 7547 1193 1054 1054
q6 178 176 152 152
q7 793 831 672 672
q8 9292 1488 1328 1328
q9 4864 4780 4762 4762
q10 6774 1886 1638 1638
q11 456 241 238 238
q12 682 568 461 461
q13 17757 4226 3424 3424
q14 223 228 210 210
q15 924 801 793 793
q16 709 729 668 668
q17 704 870 425 425
q18 6013 5395 5263 5263
q19 1236 990 646 646
q20 515 494 393 393
q21 4701 1836 1432 1432
q22 336 289 241 241
Total cold run time: 96638 ms
Total hot run time: 28900 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4409 4322 4351 4322
q2 q3 1762 2203 1738 1738
q4 881 1160 778 778
q5 4032 4316 4333 4316
q6 185 173 143 143
q7 1706 1610 1473 1473
q8 2419 2663 2520 2520
q9 7544 7496 7467 7467
q10 2707 2878 2448 2448
q11 520 442 407 407
q12 489 582 447 447
q13 3988 4453 3737 3737
q14 301 324 301 301
q15 877 831 828 828
q16 742 750 700 700
q17 1155 1518 1330 1330
q18 7145 6748 6655 6655
q19 966 982 930 930
q20 2060 2134 1973 1973
q21 3984 3513 3337 3337
q22 481 444 380 380
Total cold run time: 48353 ms
Total hot run time: 46230 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert-overwrite) skip deleted partitions before replace to avoid DdlException [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #60914:
URL: https://github.com/apache/doris/pull/60914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert-overwrite) skip deleted partitions before replace to avoid DdlException [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60914:
URL: https://github.com/apache/doris/pull/60914#issuecomment-3981771211
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert-overwrite) skip deleted partitions before replace to avoid DdlException [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #60914:
URL: https://github.com/apache/doris/pull/60914#issuecomment-3976748428
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert-overwrite) skip deleted partitions before replace to avoid DdlException [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60914:
URL: https://github.com/apache/doris/pull/60914#issuecomment-3976790243
<details>
<summary>TPC-DS: <b>Total hot run time: 184762 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 160ee55af10a7aaff412ee4c81e0034a9cca2dd2, data reload: false
query5 5240 641 509 509
query6 320 237 229 229
query7 4227 480 267 267
query8 355 253 246 246
query9 8750 2746 2729 2729
query10 555 392 341 341
query11 17002 17577 17083 17083
query12 199 135 134 134
query13 1338 501 371 371
query14 7776 3437 3141 3141
query14_1 2999 2917 2847 2847
query15 207 198 175 175
query16 1007 476 488 476
query17 1308 815 651 651
query18 2892 470 384 384
query19 228 268 274 268
query20 157 133 138 133
query21 232 145 145 145
query22 5645 5819 5633 5633
query23 17404 17149 16997 16997
query23_1 16676 16635 16639 16635
query24 7025 1607 1204 1204
query24_1 1192 1216 1205 1205
query25 530 448 402 402
query26 1232 258 144 144
query27 2747 458 285 285
query28 4499 1858 1856 1856
query29 795 558 472 472
query30 309 244 209 209
query31 871 730 652 652
query32 76 72 65 65
query33 506 340 270 270
query34 894 909 608 608
query35 628 674 609 609
query36 1104 1169 985 985
query37 133 99 82 82
query38 2971 2943 2859 2859
query39 896 865 827 827
query39_1 830 823 832 823
query40 225 150 138 138
query41 66 58 56 56
query42 109 102 99 99
query43 377 375 353 353
query44
query45 194 182 183 182
query46 949 980 602 602
query47 2087 2128 2046 2046
query48 303 315 232 232
query49 628 448 383 383
query50 687 281 209 209
query51 4049 4106 4054 4054
query52 105 107 99 99
query53 287 341 285 285
query54 291 263 251 251
query55 92 80 80 80
query56 312 306 305 305
query57 1367 1335 1270 1270
query58 286 272 273 272
query59 2612 2721 2538 2538
query60 335 337 315 315
query61 148 148 149 148
query62 607 600 533 533
query63 311 279 274 274
query64 4822 1252 994 994
query65
query66 1366 482 365 365
query67 16403 16478 16194 16194
query68
query69 388 306 280 280
query70 983 982 939 939
query71 332 308 297 297
query72 2778 2660 2421 2421
query73 539 543 323 323
query74 9971 9945 9719 9719
query75 2837 2746 2467 2467
query76 2294 1042 657 657
query77 365 392 310 310
query78 11251 11368 10721 10721
query79 1117 796 599 599
query80 876 625 543 543
query81 526 275 245 245
query82 1342 151 114 114
query83 346 260 238 238
query84 255 120 99 99
query85 924 479 453 453
query86 410 317 293 293
query87 3109 3137 3055 3055
query88 3535 2666 2649 2649
query89 441 369 345 345
query90 1799 172 169 169
query91 169 162 134 134
query92 78 73 72 72
query93 911 812 511 511
query94 504 325 301 301
query95 580 394 317 317
query96 636 514 230 230
query97 2513 2483 2417 2417
query98 237 224 226 224
query99 991 994 903 903
Total cold run time: 255092 ms
Total hot run time: 184762 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](insert-overwrite) skip deleted partitions before replace to avoid DdlException [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60914:
URL: https://github.com/apache/doris/pull/60914#issuecomment-3981771303
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org