You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "yujun777 (via GitHub)" <gi...@apache.org> on 2026/05/29 03:31:38 UTC
[PR] [fix](fe) Skip dropped columns in follower stats sync [doris]
yujun777 opened a new pull request, #63882:
URL: https://github.com/apache/doris/pull/63882
### What problem does this PR solve?
FollowerColumnSender drains queued column references on follower FEs and syncs the columns that still need analysis to the master. A queued column can become stale after DDL changes. If the table still exists but the queued column has been dropped, table.getColumn(column.colName) returns null and the sender throws a NullPointerException while reading the type.
This patch skips dropped columns before checking the column type, so the daemon does not emit periodic ERROR logs and can continue processing the remaining queued columns.
Issue Number: close #xxx
### Release note
None
### Check List (For Author)
- Test: Unit Test
- org.apache.doris.statistics.FollowerColumnSenderTest
- Behavior changed: No
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Skip dropped columns in follower stats sync [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #63882:
URL: https://github.com/apache/doris/pull/63882
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Skip dropped columns in follower stats sync [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #63882:
URL: https://github.com/apache/doris/pull/63882#issuecomment-4570224214
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Skip dropped columns in follower stats sync [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63882:
URL: https://github.com/apache/doris/pull/63882#issuecomment-4575071401
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/3)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63882_ff333368ee9336fbfb7f89b17c74aeeb4a85a01e_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63882_ff333368ee9336fbfb7f89b17c74aeeb4a85a01e_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Skip dropped columns in follower stats sync [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #63882:
URL: https://github.com/apache/doris/pull/63882#issuecomment-4589471674
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Skip dropped columns in follower stats sync [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #63882:
URL: https://github.com/apache/doris/pull/63882#issuecomment-4574142432
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Skip dropped columns in follower stats sync [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63882:
URL: https://github.com/apache/doris/pull/63882#issuecomment-4589474107
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Skip dropped columns in follower stats sync [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63882:
URL: https://github.com/apache/doris/pull/63882#issuecomment-4575228635
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/3)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63882_ff333368ee9336fbfb7f89b17c74aeeb4a85a01e_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63882_ff333368ee9336fbfb7f89b17c74aeeb4a85a01e_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Skip dropped columns in follower stats sync [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63882:
URL: https://github.com/apache/doris/pull/63882#issuecomment-4571044235
<details>
<summary>TPC-H: <b>Total hot run time: 31615 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ff333368ee9336fbfb7f89b17c74aeeb4a85a01e, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17718 4056 4003 4003
q2 q3 10757 1408 824 824
q4 4711 475 347 347
q5 7845 2298 2114 2114
q6 396 174 135 135
q7 928 780 659 659
q8 9556 1815 1664 1664
q9 7115 5003 5000 5000
q10 6444 2303 1912 1912
q11 430 273 249 249
q12 690 415 290 290
q13 18226 3427 2803 2803
q14 267 258 245 245
q15 q16 835 801 716 716
q17 1020 938 956 938
q18 6962 5777 5621 5621
q19 1189 1366 1130 1130
q20 502 414 266 266
q21 5769 2616 2393 2393
q22 429 359 306 306
Total cold run time: 101789 ms
Total hot run time: 31615 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4374 4406 4324 4324
q2 q3 4541 4913 4324 4324
q4 2076 2173 1373 1373
q5 4426 4305 5129 4305
q6 244 196 138 138
q7 1991 1871 1584 1584
q8 2438 2122 2243 2122
q9 8020 8026 7987 7987
q10 4903 4789 4319 4319
q11 559 410 418 410
q12 741 783 533 533
q13 3331 3651 2949 2949
q14 299 314 284 284
q15 q16 717 756 670 670
q17 1397 1384 1356 1356
q18 8122 7279 6732 6732
q19 1118 1132 1079 1079
q20 2225 2234 1954 1954
q21 5296 4596 4449 4449
q22 528 472 413 413
Total cold run time: 57346 ms
Total hot run time: 51305 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Skip dropped columns in follower stats sync [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63882:
URL: https://github.com/apache/doris/pull/63882#issuecomment-4570223377
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Skip dropped columns in follower stats sync [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63882:
URL: https://github.com/apache/doris/pull/63882#issuecomment-4571930953
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/3)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63882_ff333368ee9336fbfb7f89b17c74aeeb4a85a01e_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63882_ff333368ee9336fbfb7f89b17c74aeeb4a85a01e_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Skip dropped columns in follower stats sync [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #63882:
URL: https://github.com/apache/doris/pull/63882#issuecomment-4573813658
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Skip dropped columns in follower stats sync [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63882:
URL: https://github.com/apache/doris/pull/63882#issuecomment-4571135653
<details>
<summary>TPC-DS: <b>Total hot run time: 171821 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ff333368ee9336fbfb7f89b17c74aeeb4a85a01e, data reload: false
query5 4330 658 531 531
query6 329 227 203 203
query7 4224 606 346 346
query8 325 241 235 235
query9 8754 4064 4076 4064
query10 452 340 302 302
query11 5808 2595 2226 2226
query12 182 134 131 131
query13 1282 641 469 469
query14 6141 5492 5200 5200
query14_1 4533 4523 4506 4506
query15 218 208 188 188
query16 1041 482 475 475
query17 1179 747 619 619
query18 2728 491 362 362
query19 228 207 174 174
query20 139 135 129 129
query21 217 145 121 121
query22 13681 13621 13338 13338
query23 17347 16601 16289 16289
query23_1 16343 16404 16367 16367
query24 7437 1807 1379 1379
query24_1 1383 1333 1347 1333
query25 587 502 444 444
query26 1297 315 183 183
query27 2685 603 343 343
query28 4465 2040 2040 2040
query29 964 615 500 500
query30 289 238 196 196
query31 1139 1084 947 947
query32 90 75 75 75
query33 525 356 296 296
query34 1184 1199 651 651
query35 778 801 711 711
query36 1385 1374 1257 1257
query37 156 105 96 96
query38 3218 3170 3104 3104
query39 922 976 891 891
query39_1 879 862 862 862
query40 236 143 128 128
query41 65 63 62 62
query42 114 113 108 108
query43 330 329 295 295
query44
query45 211 211 202 202
query46 1070 1174 757 757
query47 2367 2371 2243 2243
query48 419 395 300 300
query49 624 496 379 379
query50 1028 356 263 263
query51 4400 4459 4290 4290
query52 106 106 94 94
query53 256 279 210 210
query54 308 278 259 259
query55 94 91 87 87
query56 327 304 297 297
query57 1431 1430 1314 1314
query58 298 281 266 266
query59 1604 1679 1486 1486
query60 314 323 318 318
query61 154 158 157 157
query62 705 656 593 593
query63 243 207 211 207
query64 2377 821 656 656
query65
query66 1650 482 360 360
query67 29673 29621 29573 29573
query68
query69 474 352 308 308
query70 1054 1039 1008 1008
query71 297 258 266 258
query72 2930 2714 2462 2462
query73 859 815 464 464
query74 5094 4998 4777 4777
query75 2720 2601 2263 2263
query76 2287 1125 773 773
query77 408 399 347 347
query78 12431 12451 11823 11823
query79 1484 1027 749 749
query80 637 536 453 453
query81 458 285 242 242
query82 1385 161 122 122
query83 350 281 254 254
query84 258 149 113 113
query85 923 562 460 460
query86 400 363 308 308
query87 3447 3393 3268 3268
query88 3596 2731 2724 2724
query89 451 401 342 342
query90 1897 186 187 186
query91 181 175 141 141
query92 77 81 79 79
query93 1475 1409 890 890
query94 556 367 315 315
query95 693 476 359 359
query96 1109 816 345 345
query97 2736 2738 2625 2625
query98 231 222 229 222
query99 1183 1154 1037 1037
Total cold run time: 254313 ms
Total hot run time: 171821 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Skip dropped columns in follower stats sync [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63882:
URL: https://github.com/apache/doris/pull/63882#issuecomment-4589473977
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org