You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morningman (via GitHub)" <gi...@apache.org> on 2026/01/16 07:43:30 UTC
[PR] [fix](hive) Fix partition values cache sync issue in Observer FE during insert [doris]
morningman opened a new pull request, #59965:
URL: https://github.com/apache/doris/pull/59965
### What problem does this PR solve?
Related PR: #58166
Problem Summary:
When Observer FE receives edit logs from Master FE after a Hive insert operation, it may miss adding new partitions to its partition values cache. This happens because:
1. When inserting into a new partition, Master FE refreshes its cache and gets the partition into its partition values cache
2. Master FE records the partition as MODIFIED (not NEW) in the edit log if it already exists in HMS
3. Observer FE processes only newPartNames when updating partition values cache, missing partitions marked as MODIFIED
Fix by merging both modifiedPartNames and newPartNames before adding to the partition values cache, ensuring Observer FE stays synchronized with Master FE.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hive) Fix partition values cache sync issue in Observer FE during insert [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #59965:
URL: https://github.com/apache/doris/pull/59965#issuecomment-3762578943
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hive) Fix partition values cache sync issue in Observer FE during insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #59965:
URL: https://github.com/apache/doris/pull/59965#issuecomment-3758604076
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hive) Fix partition values cache sync issue in Observer FE during insert [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #59965:
URL: https://github.com/apache/doris/pull/59965#issuecomment-3762611755
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hive) Fix partition values cache sync issue in Observer FE during insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #59965:
URL: https://github.com/apache/doris/pull/59965#issuecomment-3762623862
<details>
<summary>ClickBench: <b>Total hot run time: 27.45 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8c4c8fcf1bc05aea34863acca7b0ce6ceb5a0e5f, data reload: false
query1 0.05 0.05 0.05
query2 0.15 0.05 0.04
query3 0.28 0.09 0.08
query4 1.62 0.12 0.11
query5 0.27 0.25 0.25
query6 1.16 0.68 0.66
query7 0.03 0.02 0.03
query8 0.06 0.04 0.04
query9 0.58 0.50 0.50
query10 0.55 0.55 0.55
query11 0.17 0.11 0.11
query12 0.16 0.11 0.11
query13 0.62 0.60 0.59
query14 0.95 0.96 0.97
query15 0.82 0.79 0.78
query16 0.40 0.41 0.41
query17 1.08 1.10 1.07
query18 0.24 0.22 0.22
query19 2.07 1.93 1.94
query20 0.02 0.02 0.02
query21 15.44 0.28 0.15
query22 5.04 0.07 0.05
query23 15.72 0.29 0.10
query24 1.85 0.62 0.33
query25 0.09 0.09 0.09
query26 0.15 0.14 0.14
query27 0.08 0.05 0.06
query28 3.33 1.10 0.89
query29 12.60 4.11 3.32
query30 0.28 0.14 0.12
query31 2.83 0.67 0.41
query32 3.26 0.57 0.47
query33 3.09 3.00 3.07
query34 16.24 5.07 4.51
query35 4.44 4.54 4.47
query36 0.67 0.51 0.49
query37 0.13 0.07 0.07
query38 0.07 0.05 0.04
query39 0.07 0.03 0.04
query40 0.18 0.15 0.12
query41 0.10 0.04 0.03
query42 0.06 0.03 0.03
query43 0.04 0.04 0.04
Total cold run time: 97.04 s
Total hot run time: 27.45 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hive) Fix partition values cache sync issue in Observer FE during insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #59965:
URL: https://github.com/apache/doris/pull/59965#issuecomment-3762610478
<details>
<summary>TPC-H: <b>Total hot run time: 32205 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8c4c8fcf1bc05aea34863acca7b0ce6ceb5a0e5f, data reload: false
------ Round 1 ----------------------------------
q1 17715 4269 4132 4132
q2 2251 355 237 237
q3 10230 1288 747 747
q4 10243 852 320 320
q5 7516 2130 1889 1889
q6 195 171 139 139
q7 922 803 668 668
q8 9313 1461 1286 1286
q9 4884 4596 4697 4596
q10 7240 1785 1403 1403
q11 646 321 299 299
q12 730 741 618 618
q13 17784 3850 3101 3101
q14 290 300 293 293
q15 605 515 506 506
q16 708 701 646 646
q17 738 845 516 516
q18 6775 6553 6396 6396
q19 1339 1028 681 681
q20 436 399 263 263
q21 3256 2651 2468 2468
q22 1116 1066 1001 1001
Total cold run time: 104932 ms
Total hot run time: 32205 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4173 4076 4121 4076
q2 347 419 333 333
q3 2075 2602 2243 2243
q4 1360 1810 1318 1318
q5 4135 3967 4062 3967
q6 217 175 133 133
q7 2148 1963 1729 1729
q8 2579 2491 2399 2399
q9 7237 7238 7245 7238
q10 2591 2739 2357 2357
q11 564 488 455 455
q12 734 777 651 651
q13 3765 4101 3432 3432
q14 302 318 301 301
q15 557 517 498 498
q16 672 679 638 638
q17 1180 1314 1324 1314
q18 8040 7842 7928 7842
q19 953 996 922 922
q20 2053 2208 1959 1959
q21 4970 4536 4138 4138
q22 1104 1039 981 981
Total cold run time: 51756 ms
Total hot run time: 48924 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hive) Fix partition values cache sync issue in Observer FE during insert [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #59965:
URL: https://github.com/apache/doris/pull/59965#issuecomment-3758638313
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hive) Fix partition values cache sync issue in Observer FE during insert [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #59965:
URL: https://github.com/apache/doris/pull/59965#issuecomment-3762650485
# FE UT Coverage Report
Increment line coverage ` 0.00% (0/4)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/59965_8c4c8fcf1bc05aea34863acca7b0ce6ceb5a0e5f/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/59965_8c4c8fcf1bc05aea34863acca7b0ce6ceb5a0e5f/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hive) Fix partition values cache sync issue in Observer FE during insert [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #59965:
URL: https://github.com/apache/doris/pull/59965#issuecomment-3762619330
<details>
<summary>TPC-DS: <b>Total hot run time: 177343 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8c4c8fcf1bc05aea34863acca7b0ce6ceb5a0e5f, data reload: false
query5 4897 651 503 503
query6 337 248 225 225
query7 4232 488 292 292
query8 380 254 244 244
query9 8704 3195 3175 3175
query10 491 399 350 350
query11 15326 15185 14935 14935
query12 186 120 122 120
query13 1265 516 436 436
query14 6372 3114 2904 2904
query14_1 2801 2792 2796 2792
query15 217 205 181 181
query16 3268 509 479 479
query17 1069 689 569 569
query18 2228 459 349 349
query19 232 232 211 211
query20 132 123 122 122
query21 512 145 124 124
query22 4197 4281 4108 4108
query23 16218 15650 15323 15323
query23_1 15711 15463 15401 15401
query24 7229 1577 1227 1227
query24_1 1211 1219 1239 1219
query25 688 445 403 403
query26 1347 278 152 152
query27 2603 468 293 293
query28 4532 2374 2356 2356
query29 737 518 417 417
query30 472 252 212 212
query31 875 639 576 576
query32 91 78 80 78
query33 697 362 309 309
query34 919 911 559 559
query35 740 768 701 701
query36 914 929 891 891
query37 228 103 82 82
query38 3234 2744 2718 2718
query39 774 775 741 741
query39_1 705 719 728 719
query40 370 138 123 123
query41 69 71 64 64
query42 107 103 104 103
query43 493 483 488 483
query44 1505 818 810 810
query45 187 187 183 183
query46 876 972 605 605
query47 1499 1503 1365 1365
query48 353 357 259 259
query49 686 433 362 362
query50 670 293 225 225
query51 3904 3776 3822 3776
query52 144 112 101 101
query53 306 325 278 278
query54 450 269 264 264
query55 84 82 85 82
query56 310 321 318 318
query57 1076 1023 923 923
query58 277 267 272 267
query59 2343 2273 2285 2273
query60 396 362 315 315
query61 167 164 157 157
query62 419 377 325 325
query63 310 267 273 267
query64 4528 1303 1004 1004
query65 3873 3726 3787 3726
query66 1364 442 320 320
query67 15574 15644 15502 15502
query68 2461 1169 846 846
query69 458 381 335 335
query70 1085 1024 986 986
query71 344 325 299 299
query72 5309 3133 3398 3133
query73 628 702 345 345
query74 8793 8917 8602 8602
query75 2796 2858 2496 2496
query76 1844 1084 740 740
query77 423 407 342 342
query78 9820 9916 9171 9171
query79 2881 950 625 625
query80 1689 579 505 505
query81 574 267 236 236
query82 1002 156 115 115
query83 355 277 268 268
query84 279 131 104 104
query85 939 509 445 445
query86 417 342 316 316
query87 2917 2894 2867 2867
query88 3939 2847 2814 2814
query89 410 366 341 341
query90 1941 188 172 172
query91 179 171 151 151
query92 94 75 75 75
query93 1203 993 598 598
query94 671 315 287 287
query95 598 345 335 335
query96 727 538 247 247
query97 2342 2420 2331 2331
query98 241 210 206 206
query99 635 594 545 545
Total cold run time: 256318 ms
Total hot run time: 177343 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](hive) Fix partition values cache sync issue in Observer FE during insert [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #59965:
URL: https://github.com/apache/doris/pull/59965
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org