You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "sollhui (via GitHub)" <gi...@apache.org> on 2026/05/12 11:22:41 UTC
[PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
sollhui opened a new pull request, #63180:
URL: https://github.com/apache/doris/pull/63180
### What problem does this PR solve?
Problem Summary:
In a single-BE deployment, Kafka routine load fetches topic metadata through the only BE. If that BE cannot connect to Kafka, the metadata request fails and the BE is skipped in the current retry loop. Then FE may have no normal candidate backend left and falls back to backend ids in the routine load blacklist.
The blacklist can contain stale backend ids that no longer exist in `SystemInfoService`. In that case, `KafkaUtil` may get a null `Backend` and throw a NullPointerException when calling `be.getHost()`. This hides the real Kafka metadata error, such as broker connection failure.
This PR filters stale backend ids when reading the routine load blacklist and adds a final null check before creating the BE address. The original Kafka metadata error is preserved instead of being replaced by the secondary NPE.
A regression case is added with an invalid `kafka_broker_list` to verify that routine load reports the expected Kafka metadata error path.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4452231926
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4432702437
<details>
<summary>TPC-DS: <b>Total hot run time: 171056 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0e645dd499930a48b9bc297b0484aeb4b44efba5, data reload: false
query5 4299 677 529 529
query6 347 216 214 214
query7 4313 592 318 318
query8 344 245 243 243
query9 8880 4083 4044 4044
query10 464 357 328 328
query11 5790 2407 2201 2201
query12 186 129 130 129
query13 1313 613 445 445
query14 6508 5427 5105 5105
query14_1 4481 4468 4424 4424
query15 212 206 187 187
query16 1008 505 445 445
query17 1158 787 675 675
query18 2753 506 368 368
query19 232 209 169 169
query20 146 136 131 131
query21 219 147 124 124
query22 13599 13544 13335 13335
query23 17143 16373 16650 16373
query23_1 16288 16324 16407 16324
query24 7866 1826 1436 1436
query24_1 1427 1412 1490 1412
query25 654 558 503 503
query26 1575 340 174 174
query27 2927 597 353 353
query28 4411 1983 2002 1983
query29 1026 643 512 512
query30 300 234 194 194
query31 1108 1064 943 943
query32 78 70 71 70
query33 533 353 294 294
query34 1180 1117 641 641
query35 804 798 674 674
query36 1285 1303 1148 1148
query37 148 102 84 84
query38 3236 3111 3073 3073
query39 925 909 879 879
query39_1 871 882 862 862
query40 227 155 132 132
query41 63 60 59 59
query42 110 116 115 115
query43 337 329 295 295
query44
query45 212 210 189 189
query46 1066 1256 721 721
query47 2282 2219 2164 2164
query48 413 413 284 284
query49 648 535 411 411
query50 713 291 222 222
query51 4319 4307 4324 4307
query52 104 108 95 95
query53 252 280 220 220
query54 307 281 260 260
query55 93 87 83 83
query56 294 342 312 312
query57 1392 1358 1277 1277
query58 299 273 271 271
query59 1553 1634 1400 1400
query60 349 340 333 333
query61 159 155 160 155
query62 663 625 530 530
query63 246 201 204 201
query64 2371 842 707 707
query65
query66 1688 529 401 401
query67 30023 30046 29813 29813
query68
query69 468 333 311 311
query70 1037 1008 986 986
query71 325 287 276 276
query72 2890 2754 2435 2435
query73 892 768 417 417
query74 5070 4933 4711 4711
query75 2781 2669 2324 2324
query76 2285 1176 777 777
query77 420 442 350 350
query78 12845 12993 12228 12228
query79 1513 959 735 735
query80 1376 596 511 511
query81 520 286 237 237
query82 899 158 122 122
query83 322 282 261 261
query84 263 139 114 114
query85 900 506 444 444
query86 448 316 312 312
query87 3406 3348 3215 3215
query88 3636 2682 2654 2654
query89 437 379 341 341
query90 1957 189 186 186
query91 178 167 139 139
query92 83 78 69 69
query93 1153 985 553 553
query94 719 349 310 310
query95 647 472 359 359
query96 999 788 319 319
query97 2701 2698 2587 2587
query98 241 241 233 233
query99 1091 1136 966 966
Total cold run time: 255547 ms
Total hot run time: 171056 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4431550027
<details>
<summary>TPC-DS: <b>Total hot run time: 170326 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d506213350b6f45ed33e521285a05fd78d631a86, data reload: false
query5 4298 646 516 516
query6 321 223 200 200
query7 4336 569 313 313
query8 321 233 216 216
query9 8824 3957 3949 3949
query10 452 336 287 287
query11 5773 2382 2192 2192
query12 188 128 127 127
query13 1277 590 440 440
query14 6007 5367 5133 5133
query14_1 4372 4386 4332 4332
query15 213 202 186 186
query16 1020 468 460 460
query17 1171 809 653 653
query18 2456 506 364 364
query19 227 215 183 183
query20 141 143 133 133
query21 216 137 124 124
query22 13646 13600 13563 13563
query23 17133 16245 15966 15966
query23_1 16213 16011 16145 16011
query24 7424 1761 1335 1335
query24_1 1358 1372 1354 1354
query25 590 523 471 471
query26 1358 314 166 166
query27 2705 586 339 339
query28 4417 1952 1954 1952
query29 940 635 513 513
query30 303 238 200 200
query31 1111 1043 928 928
query32 89 70 70 70
query33 517 351 287 287
query34 1162 1167 635 635
query35 758 792 671 671
query36 1321 1352 1170 1170
query37 155 108 87 87
query38 3232 3129 3011 3011
query39 940 927 900 900
query39_1 869 879 868 868
query40 237 159 136 136
query41 65 63 64 63
query42 112 115 112 112
query43 316 316 279 279
query44
query45 206 196 187 187
query46 1101 1254 716 716
query47 2263 2360 2142 2142
query48 395 426 293 293
query49 651 519 416 416
query50 701 295 213 213
query51 4292 4314 4222 4222
query52 103 105 94 94
query53 252 268 225 225
query54 303 272 260 260
query55 99 88 83 83
query56 294 296 299 296
query57 1415 1367 1304 1304
query58 287 281 253 253
query59 1529 1605 1401 1401
query60 346 342 322 322
query61 157 155 159 155
query62 669 637 542 542
query63 240 202 208 202
query64 2408 812 667 667
query65
query66 1782 515 391 391
query67 29939 29205 29823 29205
query68
query69 456 336 306 306
query70 990 1012 950 950
query71 295 274 271 271
query72 2949 2709 2446 2446
query73 862 808 454 454
query74 5054 4889 4740 4740
query75 2770 2634 2325 2325
query76 2294 1147 753 753
query77 423 437 366 366
query78 13026 12972 12367 12367
query79 1471 994 758 758
query80 704 617 539 539
query81 455 281 238 238
query82 1398 164 128 128
query83 375 287 255 255
query84 276 147 118 118
query85 889 506 437 437
query86 391 345 316 316
query87 3458 3325 3204 3204
query88 3588 2680 2655 2655
query89 455 377 334 334
query90 1908 174 180 174
query91 177 171 140 140
query92 75 75 71 71
query93 953 950 552 552
query94 527 334 305 305
query95 670 456 333 333
query96 1061 785 342 342
query97 2703 2682 2549 2549
query98 238 227 230 227
query99 1145 1104 1010 1010
Total cold run time: 252803 ms
Total hot run time: 170326 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4430609417
# FE UT Coverage Report
Increment line coverage ` 0.00% (0/17)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63180_d506213350b6f45ed33e521285a05fd78d631a86/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63180_d506213350b6f45ed33e521285a05fd78d631a86/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4436539247
# FE UT Coverage Report
Increment line coverage ` 0.00% (0/15)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63180_0e645dd499930a48b9bc297b0484aeb4b44efba5/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63180_0e645dd499930a48b9bc297b0484aeb4b44efba5/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4432339984
<details>
<summary>TPC-H: <b>Total hot run time: 29473 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 13e920a3e5193eaa6c3216de34ff03f6aae465dd, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17676 3903 3837 3837
q2 q3 10715 883 622 622
q4 4671 468 348 348
q5 7483 1338 1159 1159
q6 189 170 143 143
q7 927 970 778 778
q8 9351 1361 1331 1331
q9 5576 5336 5280 5280
q10 6318 2083 1795 1795
q11 489 270 267 267
q12 700 411 294 294
q13 18220 3363 2718 2718
q14 297 285 265 265
q15 q16 908 865 781 781
q17 986 1033 763 763
q18 6274 5668 5472 5472
q19 1184 1240 1057 1057
q20 502 390 259 259
q21 4953 2354 1974 1974
q22 469 410 330 330
Total cold run time: 97888 ms
Total hot run time: 29473 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4571 4517 4497 4497
q2 q3 4640 4736 4223 4223
q4 2119 2140 1377 1377
q5 4999 5015 5236 5015
q6 206 174 133 133
q7 2066 1787 1617 1617
q8 3381 3121 3133 3121
q9 8437 8651 8376 8376
q10 4465 4503 4297 4297
q11 594 434 431 431
q12 728 758 554 554
q13 3297 3607 2923 2923
q14 307 321 289 289
q15 q16 781 792 704 704
q17 1331 1282 1250 1250
q18 7919 7149 7105 7105
q19 1194 1172 1163 1163
q20 2228 2214 1915 1915
q21 5992 5298 4742 4742
q22 514 468 401 401
Total cold run time: 59769 ms
Total hot run time: 54133 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4431456477
<details>
<summary>TPC-H: <b>Total hot run time: 29587 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d506213350b6f45ed33e521285a05fd78d631a86, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17625 3997 3914 3914
q2 q3 10719 868 606 606
q4 4660 459 340 340
q5 7467 1359 1157 1157
q6 201 178 144 144
q7 920 965 748 748
q8 9328 1437 1276 1276
q9 5498 5330 5299 5299
q10 6269 2052 1844 1844
q11 475 267 263 263
q12 629 422 302 302
q13 18084 3401 2647 2647
q14 296 283 266 266
q15 q16 866 875 788 788
q17 1005 1170 708 708
q18 6491 5717 5621 5621
q19 1225 1337 1083 1083
q20 516 405 272 272
q21 4537 2409 1959 1959
q22 485 422 350 350
Total cold run time: 97296 ms
Total hot run time: 29587 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4885 4726 4889 4726
q2 q3 4669 4839 4182 4182
q4 2160 2169 1392 1392
q5 5020 4967 5234 4967
q6 214 181 147 147
q7 2028 1804 1623 1623
q8 3358 3113 3099 3099
q9 8420 8482 8325 8325
q10 4523 4502 4264 4264
q11 600 405 390 390
q12 685 751 521 521
q13 3263 3587 2892 2892
q14 386 392 286 286
q15 q16 784 778 706 706
q17 1383 1345 1295 1295
q18 8127 7104 7186 7104
q19 1174 1155 1160 1155
q20 2210 2225 1959 1959
q21 6157 5464 4984 4984
q22 533 519 413 413
Total cold run time: 60579 ms
Total hot run time: 54430 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 merged PR #63180:
URL: https://github.com/apache/doris/pull/63180
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4432622419
<details>
<summary>TPC-H: <b>Total hot run time: 29537 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0e645dd499930a48b9bc297b0484aeb4b44efba5, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17789 4030 3973 3973
q2 q3 10725 865 614 614
q4 4661 449 354 354
q5 7449 1321 1151 1151
q6 205 179 145 145
q7 927 957 749 749
q8 9379 1419 1279 1279
q9 5627 5340 5285 5285
q10 6287 2077 1810 1810
q11 477 260 257 257
q12 687 406 288 288
q13 18192 3308 2708 2708
q14 289 284 263 263
q15 q16 900 860 788 788
q17 979 1008 785 785
q18 6401 5695 5606 5606
q19 1177 1259 1049 1049
q20 532 394 261 261
q21 4563 2266 1873 1873
q22 416 350 299 299
Total cold run time: 97662 ms
Total hot run time: 29537 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4181 4193 4232 4193
q2 q3 4632 4754 4195 4195
q4 2088 2143 1360 1360
q5 4958 4887 5222 4887
q6 193 171 133 133
q7 2060 1800 1951 1800
q8 3448 3208 3211 3208
q9 8439 8449 8418 8418
q10 4483 4498 4273 4273
q11 623 464 408 408
q12 694 745 525 525
q13 3175 3558 2921 2921
q14 308 312 282 282
q15 q16 803 812 713 713
q17 1380 1366 1303 1303
q18 8281 7233 7213 7213
q19 1149 1153 1184 1153
q20 2270 2286 1949 1949
q21 6191 5490 4873 4873
q22 554 517 427 427
Total cold run time: 59910 ms
Total hot run time: 54234 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4432980916
# FE UT Coverage Report
Increment line coverage ` 0.00% (0/15)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63180_0e645dd499930a48b9bc297b0484aeb4b44efba5/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63180_0e645dd499930a48b9bc297b0484aeb4b44efba5/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4429997987
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4430682773
<details>
<summary>TPC-H: <b>Total hot run time: 29521 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d506213350b6f45ed33e521285a05fd78d631a86, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17617 3910 3894 3894
q2 q3 10722 885 620 620
q4 4655 457 349 349
q5 7431 1353 1138 1138
q6 208 174 145 145
q7 945 946 764 764
q8 9706 1401 1288 1288
q9 6273 5436 5317 5317
q10 6336 2090 1782 1782
q11 475 275 258 258
q12 687 421 303 303
q13 18243 3328 2807 2807
q14 305 285 262 262
q15 q16 929 863 790 790
q17 990 1051 699 699
q18 6366 5723 5597 5597
q19 1228 1278 1050 1050
q20 521 414 270 270
q21 4558 2291 1878 1878
q22 425 360 310 310
Total cold run time: 98620 ms
Total hot run time: 29521 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4280 4097 4046 4046
q2 q3 4642 4781 4171 4171
q4 2085 2159 1379 1379
q5 4965 4938 5153 4938
q6 188 165 135 135
q7 2056 2032 1708 1708
q8 3451 3223 3202 3202
q9 8553 8434 8402 8402
q10 4503 4548 4240 4240
q11 625 438 433 433
q12 718 773 522 522
q13 3187 3617 2886 2886
q14 305 309 276 276
q15 q16 791 808 693 693
q17 1388 1341 1323 1323
q18 8043 7055 7140 7055
q19 1191 1180 1168 1168
q20 2287 2263 1970 1970
q21 6151 5416 4899 4899
q22 582 529 449 449
Total cold run time: 59991 ms
Total hot run time: 53895 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4432236701
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4432043032
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4432434194
<details>
<summary>TPC-DS: <b>Total hot run time: 170235 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 13e920a3e5193eaa6c3216de34ff03f6aae465dd, data reload: false
query5 4316 671 519 519
query6 336 225 208 208
query7 4226 582 299 299
query8 330 237 224 224
query9 8852 4073 3977 3977
query10 490 344 306 306
query11 5770 2417 2175 2175
query12 188 135 130 130
query13 1340 608 426 426
query14 5951 5332 5055 5055
query14_1 4374 4361 4337 4337
query15 217 204 188 188
query16 1006 485 449 449
query17 1155 762 643 643
query18 2738 498 354 354
query19 223 211 165 165
query20 139 135 129 129
query21 223 140 126 126
query22 13515 13544 13275 13275
query23 17049 16401 16033 16033
query23_1 16048 16257 16137 16137
query24 7537 1764 1335 1335
query24_1 1384 1357 1369 1357
query25 591 515 484 484
query26 1305 313 175 175
query27 2694 582 344 344
query28 4447 1967 1940 1940
query29 1039 657 546 546
query30 320 248 199 199
query31 1124 1062 943 943
query32 88 85 78 78
query33 540 356 308 308
query34 1151 1082 635 635
query35 755 786 694 694
query36 1294 1332 1143 1143
query37 150 104 92 92
query38 3174 3141 3028 3028
query39 918 932 910 910
query39_1 868 891 879 879
query40 228 154 135 135
query41 67 65 62 62
query42 110 114 112 112
query43 323 322 283 283
query44
query45 208 206 194 194
query46 1088 1137 747 747
query47 2324 2384 2253 2253
query48 416 413 300 300
query49 637 550 460 460
query50 737 298 225 225
query51 4365 4300 4273 4273
query52 107 108 94 94
query53 252 290 205 205
query54 311 271 255 255
query55 91 92 83 83
query56 302 314 306 306
query57 1408 1395 1337 1337
query58 289 274 271 271
query59 1495 1596 1336 1336
query60 353 338 321 321
query61 163 158 163 158
query62 671 644 572 572
query63 247 201 210 201
query64 2420 821 688 688
query65
query66 1680 523 387 387
query67 29961 29288 29829 29288
query68
query69 450 336 301 301
query70 1001 996 955 955
query71 317 283 266 266
query72 2997 2741 2401 2401
query73 832 803 444 444
query74 5075 4864 4723 4723
query75 2740 2663 2318 2318
query76 2263 1131 754 754
query77 429 441 337 337
query78 12886 12973 12296 12296
query79 1474 942 761 761
query80 1397 575 477 477
query81 528 277 247 247
query82 1178 154 120 120
query83 354 279 245 245
query84 263 137 107 107
query85 921 513 445 445
query86 436 327 324 324
query87 3465 3345 3225 3225
query88 3526 2658 2648 2648
query89 446 381 339 339
query90 2025 178 176 176
query91 175 173 140 140
query92 80 78 72 72
query93 972 963 551 551
query94 736 341 301 301
query95 680 477 345 345
query96 1031 762 354 354
query97 2724 2692 2577 2577
query98 244 232 229 229
query99 1131 1121 991 991
Total cold run time: 253792 ms
Total hot run time: 170235 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4430769769
<details>
<summary>TPC-DS: <b>Total hot run time: 173101 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d506213350b6f45ed33e521285a05fd78d631a86, data reload: false
query5 4340 662 531 531
query6 334 220 200 200
query7 4357 553 321 321
query8 320 227 212 212
query9 8801 4059 4048 4048
query10 462 344 302 302
query11 5954 2421 2235 2235
query12 181 130 126 126
query13 1297 594 452 452
query14 6850 5380 5080 5080
query14_1 4414 4366 4332 4332
query15 210 201 181 181
query16 1033 459 398 398
query17 1116 746 610 610
query18 2743 481 346 346
query19 209 193 165 165
query20 138 132 139 132
query21 228 151 117 117
query22 13655 14149 14429 14149
query23 17486 16454 16333 16333
query23_1 16275 16332 16257 16257
query24 7625 1733 1379 1379
query24_1 1362 1335 1373 1335
query25 548 479 440 440
query26 1057 326 163 163
query27 2677 594 329 329
query28 4359 1957 1969 1957
query29 977 623 511 511
query30 303 237 195 195
query31 1112 1060 924 924
query32 83 70 73 70
query33 534 350 296 296
query34 1175 1091 631 631
query35 762 785 660 660
query36 1344 1353 1160 1160
query37 144 100 88 88
query38 3216 3120 3091 3091
query39 924 917 904 904
query39_1 869 879 859 859
query40 238 160 140 140
query41 105 65 58 58
query42 108 107 104 104
query43 331 326 290 290
query44
query45 210 199 189 189
query46 1084 1228 717 717
query47 2349 2287 2216 2216
query48 400 414 304 304
query49 640 527 428 428
query50 705 292 220 220
query51 4250 4285 4252 4252
query52 117 104 95 95
query53 262 286 203 203
query54 318 270 247 247
query55 92 88 90 88
query56 319 304 308 304
query57 1402 1364 1279 1279
query58 304 274 267 267
query59 1573 1618 1430 1430
query60 344 336 326 326
query61 161 153 155 153
query62 680 604 578 578
query63 248 200 200 200
query64 2024 825 725 725
query65
query66 1646 494 395 395
query67 29392 29920 29794 29794
query68
query69 457 330 314 314
query70 1039 987 1010 987
query71 304 286 280 280
query72 2855 2709 2436 2436
query73 855 768 435 435
query74 5090 4927 4749 4749
query75 2805 2663 2353 2353
query76 2321 1137 795 795
query77 433 442 367 367
query78 12858 12829 12405 12405
query79 1479 997 752 752
query80 717 649 511 511
query81 459 289 248 248
query82 1239 167 125 125
query83 356 297 259 259
query84 273 149 117 117
query85 920 591 531 531
query86 417 347 320 320
query87 3426 3381 3246 3246
query88 3618 2676 2659 2659
query89 438 381 336 336
query90 1855 184 180 180
query91 180 165 134 134
query92 78 72 69 69
query93 951 958 559 559
query94 532 331 312 312
query95 646 376 437 376
query96 1083 735 341 341
query97 2677 2669 2597 2597
query98 244 237 234 234
query99 1114 1085 974 974
Total cold run time: 253385 ms
Total hot run time: 173101 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63180:
URL: https://github.com/apache/doris/pull/63180#discussion_r3251652462
##########
regression-test/suites/load_p0/routine_load/test_black_list.groovy:
##########
@@ -150,5 +150,61 @@ suite("test_black_list","nonConcurrent,p0") {
GetDebugPoint().disableDebugPointForAllBEs(inject)
sql "stop routine load for ${job}"
}
+
+ def invalidBrokerTableName = "test_black_list_invalid_broker"
+ def invalidBrokerJob = "test_black_list_invalid_broker_job"
+ sql """ DROP TABLE IF EXISTS ${invalidBrokerTableName} """
+ sql """
+ CREATE TABLE IF NOT EXISTS ${invalidBrokerTableName} (
+ `k1` int(20) NULL,
+ `k2` string NULL,
+ `v1` date NULL,
+ `v2` string NULL,
+ `v3` datetime NULL,
+ `v4` string NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`k1`)
+ COMMENT 'OLAP'
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+
+ try {
+ sql """
+ CREATE ROUTINE LOAD ${invalidBrokerJob} ON ${invalidBrokerTableName}
+ COLUMNS TERMINATED BY ","
+ FROM KAFKA
+ (
+ "kafka_broker_list" = "127.0.0.1:1",
Review Comment:
This regression does not exercise the stale-blacklist case fixed in KafkaUtil. The reported NPE requires `RoutineLoadManager`'s blacklist to contain a BE id that no longer exists in `SystemInfoService`, then a real metadata failure causes `getInfoRequest` to fall back to blacklist entries. This test only points an existing BE at an invalid broker and never inserts/removes a BE to leave a stale blacklist id, so it would still pass if the old code blindly used `getBlacklist().keySet()` in a normal environment. Please add setup that creates a stale blacklist entry, or add a focused FE unit test around this selection path, and assert that the original Kafka metadata error is preserved instead of an NPE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4464727526
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4452232228
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4429998884
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](job) fix NPE in routine load Kafka meta request [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63180:
URL: https://github.com/apache/doris/pull/63180#issuecomment-4433553708
# FE Regression Coverage Report
Increment line coverage ` 20.00% (3/15)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63180_0e645dd499930a48b9bc297b0484aeb4b44efba5_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63180_0e645dd499930a48b9bc297b0484aeb4b44efba5_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org