You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zclllyybb (via GitHub)" <gi...@apache.org> on 2025/05/07 08:37:02 UTC
[PR] [Fix](iot-auto-detect) Fix insert overwrite auto detect transaction concurrent conflict [doris]
zclllyybb opened a new pull request, #50658:
URL: https://github.com/apache/doris/pull/50658
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Fix bug of:
1.
```
Caused by: java.util.ConcurrentModificationException
at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1486) ~[?:?]
at java.util.TreeMap$KeyIterator.next(TreeMap.java:1540) ~[?:?]
at java.util.AbstractCollection.addAll(AbstractCollection.java:335) ~[?:?]
at java.util.HashSet.<init>(HashSet.java:121) ~[?:?]
at com.google.common.collect.Sets.newHashSet(Sets.java:226) ~[guava-33.2.1-jre.jar:?]
at org.apache.doris.catalog.OlapTable.getPartitionNames(OlapTable.java:1288) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.nereids.trees.plans.commands.insert.InsertOverwriteTableCommand.run(InsertOverwriteTableCommand.java:164) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:767) ~[doris-fe.jar:1.2-SNAPSHOT]
... 12 more
```
2.
```
2025-04-11 03:25:13,208 ERROR (thrift-server-pool-5|171) [ProcessFunction.process():47] Internal error processing replacePartition
java.lang.reflect.UndeclaredThrowableException: null
at jdk.proxy2.$Proxy44.replacePartition(Unknown Source) ~[?:?]
at org.apache.doris.thrift.FrontendService$Processor$replacePartition.getResult(FrontendService.java:5118) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT]
at org.apache.doris.thrift.FrontendService$Processor$replacePartition.getResult(FrontendService.java:5098) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.16.0.jar:0.16.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[libthrift-0.16.0.jar:0.16.0]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[libthrift-0.16.0.jar:0.16.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at org.apache.doris.service.FeServer.lambda$start$0(FeServer.java:60) ~[doris-fe.jar:1.2-SNAPSHOT]
... 9 more
Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.doris.catalog.Partition.getName()" because the return value of "org.apache.doris.catalog.OlapTable.getPartition(long)" is null
at org.apache.doris.catalog.OlapTable.getEqualPartitionNames(OlapTable.java:1296) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.service.FrontendServiceImpl.replacePartition(FrontendServiceImpl.java:3787) ~[doris-fe.jar:1.2-SNAPSHOT]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at org.apache.doris.service.FeServer.lambda$start$0(FeServer.java:60) ~[doris-fe.jar:1.2-SNAPSHOT]
... 9 more
```
### Release note
Fix unexpected failure when insert overwrite auto detect conflict with other partition operations
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [x] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [x] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](iot-auto-detect) Fix insert overwrite auto detect transaction concurrent conflict [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt merged PR #50658:
URL: https://github.com/apache/doris/pull/50658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](iot-auto-detect) Fix insert overwrite auto detect transaction concurrent conflict [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50658:
URL: https://github.com/apache/doris/pull/50658#issuecomment-2857678861
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](iot-auto-detect) Fix insert overwrite auto detect transaction concurrent conflict [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50658:
URL: https://github.com/apache/doris/pull/50658#issuecomment-2861523611
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](iot-auto-detect) Fix insert overwrite auto detect transaction concurrent conflict [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50658:
URL: https://github.com/apache/doris/pull/50658#issuecomment-2857859102
<details>
<summary>ClickBench: <b>Total hot run time: 29.85 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 392cefe2f38d3231733db4a76b2142623cf85afa, data reload: false
query1 0.03 0.05 0.02
query2 0.12 0.10 0.11
query3 0.25 0.19 0.20
query4 1.59 0.19 0.18
query5 0.59 0.57 0.61
query6 1.20 0.71 0.73
query7 0.02 0.02 0.01
query8 0.04 0.04 0.04
query9 0.57 0.52 0.51
query10 0.56 0.57 0.56
query11 0.16 0.11 0.10
query12 0.16 0.11 0.11
query13 0.61 0.59 0.59
query14 0.78 0.82 0.79
query15 0.88 0.86 0.86
query16 0.38 0.40 0.37
query17 1.01 1.05 1.05
query18 0.21 0.19 0.20
query19 1.88 1.84 1.84
query20 0.01 0.02 0.01
query21 15.39 0.89 0.55
query22 0.73 1.05 0.90
query23 14.85 1.37 0.64
query24 7.37 1.20 1.49
query25 0.49 0.17 0.12
query26 0.66 0.16 0.13
query27 0.05 0.05 0.06
query28 9.39 0.88 0.43
query29 12.56 4.04 3.33
query30 0.25 0.09 0.07
query31 2.89 0.60 0.39
query32 3.59 0.54 0.46
query33 3.05 3.00 3.13
query34 15.73 5.12 4.49
query35 4.49 4.49 4.50
query36 0.67 0.50 0.48
query37 0.09 0.07 0.06
query38 0.04 0.04 0.04
query39 0.03 0.03 0.02
query40 0.17 0.13 0.13
query41 0.08 0.03 0.02
query42 0.04 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 103.69 s
Total hot run time: 29.85 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](iot-auto-detect) Fix insert overwrite auto detect transaction concurrent conflict [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50658:
URL: https://github.com/apache/doris/pull/50658#issuecomment-2857839525
<details>
<summary>TPC-DS: <b>Total hot run time: 191732 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 392cefe2f38d3231733db4a76b2142623cf85afa, data reload: false
query1 1394 1086 1053 1053
query2 6317 1738 1737 1737
query3 11020 4453 4548 4453
query4 25588 23880 22961 22961
query5 4378 601 455 455
query6 286 204 214 204
query7 3995 486 282 282
query8 311 257 248 248
query9 8485 2549 2541 2541
query10 477 331 267 267
query11 15353 15234 14876 14876
query12 164 110 107 107
query13 1572 520 406 406
query14 9137 6120 6018 6018
query15 196 185 175 175
query16 7538 624 515 515
query17 1178 703 572 572
query18 2060 421 322 322
query19 196 193 156 156
query20 131 124 119 119
query21 210 130 108 108
query22 4517 4619 4565 4565
query23 34583 33636 33593 33593
query24 8444 2409 2395 2395
query25 503 449 388 388
query26 1207 274 157 157
query27 2839 505 331 331
query28 4757 2133 2095 2095
query29 709 571 438 438
query30 277 229 189 189
query31 885 856 785 785
query32 72 66 63 63
query33 561 359 318 318
query34 800 869 523 523
query35 805 852 741 741
query36 973 1016 929 929
query37 119 101 76 76
query38 4200 4178 4190 4178
query39 1493 1416 1422 1416
query40 216 125 105 105
query41 54 53 52 52
query42 129 110 112 110
query43 495 498 462 462
query44 1312 824 831 824
query45 184 170 170 170
query46 832 1039 655 655
query47 1870 1920 1831 1831
query48 380 408 312 312
query49 742 514 443 443
query50 653 681 402 402
query51 4329 4258 4207 4207
query52 106 115 99 99
query53 217 253 179 179
query54 573 615 517 517
query55 82 83 84 83
query56 307 300 292 292
query57 1182 1229 1144 1144
query58 260 259 245 245
query59 2691 2737 2654 2654
query60 343 326 308 308
query61 132 126 128 126
query62 785 756 681 681
query63 229 198 190 190
query64 4147 1155 785 785
query65 4462 4296 4349 4296
query66 1188 407 307 307
query67 15994 15615 15556 15556
query68 8614 880 509 509
query69 478 301 256 256
query70 1224 1114 1121 1114
query71 469 321 289 289
query72 5535 4647 4697 4647
query73 691 574 341 341
query74 9103 9257 8611 8611
query75 3856 3203 2686 2686
query76 3682 1198 752 752
query77 782 366 272 272
query78 9970 10106 9144 9144
query79 1635 815 563 563
query80 670 546 434 434
query81 494 256 224 224
query82 419 123 93 93
query83 271 247 230 230
query84 283 111 82 82
query85 761 353 312 312
query86 330 312 295 295
query87 4430 4450 4321 4321
query88 2709 2166 2143 2143
query89 401 311 286 286
query90 1920 207 204 204
query91 141 134 109 109
query92 77 57 59 57
query93 1152 944 577 577
query94 678 411 302 302
query95 378 286 279 279
query96 485 559 268 268
query97 3158 3243 3078 3078
query98 220 213 198 198
query99 1417 1399 1266 1266
Total cold run time: 278057 ms
Total hot run time: 191732 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](iot-auto-detect) Fix insert overwrite auto detect transaction concurrent conflict [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50658:
URL: https://github.com/apache/doris/pull/50658#issuecomment-2861523656
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](iot-auto-detect) Fix insert overwrite auto detect transaction concurrent conflict [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #50658:
URL: https://github.com/apache/doris/pull/50658#issuecomment-2857678994
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](iot-auto-detect) Fix insert overwrite auto detect transaction concurrent conflict [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50658:
URL: https://github.com/apache/doris/pull/50658#issuecomment-2857799754
<details>
<summary>TPC-H: <b>Total hot run time: 34812 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 392cefe2f38d3231733db4a76b2142623cf85afa, data reload: false
------ Round 1 ----------------------------------
q1 25660 5066 5017 5017
q2 2064 292 181 181
q3 10384 1261 695 695
q4 10227 985 515 515
q5 7530 2285 2377 2285
q6 181 163 138 138
q7 913 782 625 625
q8 9324 1278 1197 1197
q9 6809 5066 5143 5066
q10 6797 2301 1892 1892
q11 481 287 264 264
q12 343 389 216 216
q13 17773 3688 3134 3134
q14 218 234 225 225
q15 523 480 470 470
q16 413 433 377 377
q17 594 853 355 355
q18 7525 7099 7178 7099
q19 1215 957 595 595
q20 339 340 219 219
q21 4056 3412 3239 3239
q22 1066 1041 1008 1008
Total cold run time: 114435 ms
Total hot run time: 34812 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5105 5099 5071 5071
q2 247 328 228 228
q3 2262 2724 2284 2284
q4 1336 1754 1370 1370
q5 4491 4393 4423 4393
q6 217 185 132 132
q7 2031 1956 1830 1830
q8 2579 2486 2566 2486
q9 7192 7162 7127 7127
q10 2966 3176 2722 2722
q11 584 500 490 490
q12 689 767 631 631
q13 3482 3835 3290 3290
q14 308 302 281 281
q15 542 524 503 503
q16 434 487 452 452
q17 1143 1513 1379 1379
q18 7704 7505 7365 7365
q19 810 842 978 842
q20 2041 1972 1785 1785
q21 5208 4799 4826 4799
q22 1070 1030 1023 1023
Total cold run time: 52441 ms
Total hot run time: 50483 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org