You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "gavinchou (via GitHub)" <gi...@apache.org> on 2026/10/23 08:08:47 UTC
[PR] [fix](load) Enforce auth for group commit stream load [doris]
gavinchou opened a new pull request, #69516:
URL: https://github.com/apache/doris/pull/69516
### What problem does this PR solve?
Issue Number: None
Problem Summary:
Explicit group commit Stream Load skips the normal transaction begin path. FE planning therefore needs to apply the same credential and target-table LOAD checks before follower forwarding or local planning.
### Release note
Enforce authentication and target-table LOAD authorization for explicit group commit Stream Load.
### Check List (For Author)
- Test
- [x] Unit Test
- `sh run-fe-ut.sh --run "org.apache.doris.service.FrontendServiceImplTest"`
- `sh build.sh --fe`
- Behavior changed:
- [x] Yes. Explicit group commit Stream Load now rejects invalid credentials, invalid tokens, or callers without target-table LOAD privilege.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) Enforce auth for group commit stream load [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou merged PR #69516:
URL: https://github.com/apache/doris/pull/69516
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) Enforce auth for group commit stream load [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69516:
URL: https://github.com/apache/doris/pull/69516#issuecomment-6366052728
<details>
<summary>TPC-DS: <b>Total hot run time: 150025 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 92cccd54cc8976bdf50a6d3af1403c1b35a9ec29, data reload: false
query5 4316 595 418 418
query6 445 215 191 191
query7 4803 438 229 229
query8 328 182 151 151
query9 8768 3046 3067 3046
query10 509 297 257 257
query11 6011 3862 3584 3584
query12 164 98 90 90
query13 1252 427 331 331
query14 6044 3911 3629 3629
query14_1 3433 3405 3395 3395
query15 215 200 181 181
query16 937 435 416 416
query17 865 652 538 538
query18 2423 425 306 306
query19 198 169 131 131
query20 85 84 86 84
query21 216 125 107 107
query22 13084 12860 12760 12760
query23 13628 13159 12415 12415
query23_1 12523 12508 12621 12508
query24 7421 818 462 462
query24_1 468 478 470 470
query25 537 433 365 365
query26 1273 262 143 143
query27 2766 435 269 269
query28 4632 1892 1926 1892
query29 1539 601 448 448
query30 308 215 185 185
query31 1005 939 815 815
query32 177 94 94 94
query33 531 306 238 238
query34 899 861 502 502
query35 742 754 650 650
query36 847 828 719 719
query37 143 100 84 84
query38 1798 1721 1661 1661
query39 818 807 767 767
query39_1 726 729 722 722
query40 219 114 96 96
query41 73 69 68 68
query42 88 84 80 80
query43 339 364 309 309
query44 1309 672 671 671
query45 216 172 164 164
query46 853 959 565 565
query47 2841 2964 2702 2702
query48 282 313 217 217
query49 576 414 292 292
query50 690 268 216 216
query51 10646 10476 10555 10476
query52 78 82 68 68
query53 190 202 148 148
query54 231 197 181 181
query55 72 66 61 61
query56 251 204 211 204
query57 1637 1614 1567 1567
query58 330 262 250 250
query59 2119 2180 1947 1947
query60 290 244 221 221
query61 149 149 147 147
query62 391 328 281 281
query63 181 155 157 155
query64 2695 971 746 746
query65 3327 3258 3258 3258
query66 1766 428 291 291
query67 20058 20024 19831 19831
query68 3011 980 597 597
query69 388 282 245 245
query70 825 771 773 771
query71 290 233 210 210
query72 2797 2557 2238 2238
query73 522 518 293 293
query74 4625 4499 4335 4335
query75 2332 2386 1890 1890
query76 2304 995 580 580
query77 334 362 271 271
query78 10015 9756 9140 9140
query79 995 845 518 518
query80 1244 425 334 334
query81 604 322 270 270
query82 970 128 101 101
query83 309 192 178 178
query84 327 113 95 95
query85 870 409 376 376
query86 423 234 230 230
query87 1990 1947 1767 1767
query88 3725 2684 2643 2643
query89 347 278 241 241
query90 1923 165 164 164
query91 162 154 133 133
query92 95 108 86 86
query93 997 987 570 570
query94 785 307 274 274
query95 566 323 362 323
query96 655 473 232 232
query97 2440 2444 2307 2307
query98 161 150 156 150
query99 773 766 662 662
Total cold run time: 236374 ms
Total hot run time: 150025 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) Enforce auth for group commit stream load [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69516:
URL: https://github.com/apache/doris/pull/69516#issuecomment-6365859922
<details>
<summary>TPC-H: <b>Total hot run time: 28641 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 92cccd54cc8976bdf50a6d3af1403c1b35a9ec29, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17586 5135 5085 5085
q2 2140 350 249 249
q3 10193 1401 828 828
q4 4683 462 334 334
q5 7452 804 524 524
q6 182 184 151 151
q7 749 803 603 603
q8 9260 1330 1063 1063
q9 5494 4192 4175 4175
q10 6824 1283 1029 1029
q11 432 249 237 237
q12 656 392 285 285
q13 18067 3667 2987 2987
q14 274 273 252 252
q15 q16 737 708 651 651
q17 1202 757 573 573
q18 7119 6188 6117 6117
q19 1108 944 579 579
q20 383 329 229 229
q21 5662 2517 2421 2421
q22 375 340 269 269
Total cold run time: 100578 ms
Total hot run time: 28641 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5944 5915 6100 5915
q2 716 621 573 573
q3 5524 5641 5386 5386
q4 2029 2152 1556 1556
q5 5794 5613 5741 5613
q6 237 187 140 140
q7 2270 1969 1819 1819
q8 3031 2671 2627 2627
q9 8027 7916 7736 7736
q10 3619 3594 3196 3196
q11 577 407 378 378
q12 652 679 493 493
q13 3308 3559 3014 3014
q14 292 284 260 260
q15 q16 662 681 611 611
q17 7648 6989 6828 6828
q18 12982 12153 12932 12153
q19 913 768 776 768
q20 2140 2147 1911 1911
q21 5499 4551 4664 4551
q22 471 456 404 404
Total cold run time: 72335 ms
Total hot run time: 65932 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) Enforce auth for group commit stream load [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69516:
URL: https://github.com/apache/doris/pull/69516#issuecomment-6360462919
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) Enforce auth for group commit stream load [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on PR #69516:
URL: https://github.com/apache/doris/pull/69516#issuecomment-6360462884
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) Enforce auth for group commit stream load [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69516:
URL: https://github.com/apache/doris/pull/69516#issuecomment-6366151961
<details>
<summary>ClickBench: <b>Total hot run time: 24.15 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 92cccd54cc8976bdf50a6d3af1403c1b35a9ec29, data reload: false
query1 0.01 0.01 0.00
query2 0.10 0.06 0.05
query3 0.27 0.09 0.09
query4 1.61 0.10 0.10
query5 0.23 0.24 0.25
query6 1.16 0.69 0.66
query7 0.04 0.01 0.01
query8 0.06 0.04 0.04
query9 0.40 0.36 0.35
query10 0.56 0.61 0.55
query11 0.19 0.14 0.14
query12 0.18 0.15 0.15
query13 0.48 0.49 0.49
query14 1.00 1.01 1.01
query15 0.62 0.62 0.60
query16 0.32 0.31 0.33
query17 1.10 1.13 1.12
query18 0.21 0.18 0.18
query19 2.08 2.03 1.92
query20 0.02 0.02 0.01
query21 15.50 0.18 0.15
query22 4.89 0.05 0.05
query23 16.32 0.33 0.12
query24 2.84 0.53 0.36
query25 0.12 0.06 0.04
query26 0.78 0.20 0.15
query27 0.06 0.06 0.03
query28 4.06 0.85 0.34
query29 12.47 4.16 3.24
query30 0.29 0.13 0.10
query31 2.76 0.56 0.33
query32 3.22 0.59 0.52
query33 3.13 3.23 3.23
query34 15.37 3.98 3.38
query35 3.33 3.31 3.33
query36 0.55 0.43 0.43
query37 0.10 0.07 0.06
query38 0.05 0.04 0.04
query39 0.03 0.03 0.03
query40 0.18 0.14 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.02
query43 0.05 0.04 0.03
Total cold run time: 96.87 s
Total hot run time: 24.15 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) Enforce auth for group commit stream load [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on PR #69516:
URL: https://github.com/apache/doris/pull/69516#issuecomment-6360463607
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) Enforce auth for group commit stream load [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69516:
URL: https://github.com/apache/doris/pull/69516#discussion_r4357585450
##########
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##########
@@ -3052,21 +3072,22 @@ public TStreamLoadPutResult streamLoadPut(TStreamLoadPutRequest request) {
}
String groupCommitMode = request.getGroupCommitMode();
- if (groupCommitMode != null && !groupCommitMode.equals("off_mode") && Env.getCurrentEnv().isMaster()
- && Config.enable_forward_group_commit_stream_load_to_follower) {
- TStreamLoadPutResult result = forwardGroupCommitStreamLoad(request);
- if (result != null) {
- return result;
- }
- }
-
TStreamLoadPutResult result = new TStreamLoadPutResult();
TStatus status = new TStatus(TStatusCode.OK);
result.setStatus(status);
StreamLoadHandler streamLoadHandler = new StreamLoadHandler(request, null, result, clientAddr);
try {
+ checkGroupCommitStreamLoadAuth(request);
Review Comment:
[P2] Fence follower replay before rechecking load credentials
With `enable_forward_group_commit_stream_load_to_follower` enabled in a non-cloud cluster, change an existing load user's password on the master and submit an explicit sync/async group-commit load using the new password before a live follower replays that edit. The master passes this check, but the forwarded `streamLoadPut` runs it again against the follower's old password and returns `ANALYSIS_ERROR`. Follower selection checks only `isAlive()`, and `forwardGroupCommitStreamLoad` returns application errors directly, so local fallback never runs. Previously this ordinary follower planning path did not authenticate again. Fence follower replay against the master's auth metadata, or safely retry the master-authenticated request locally on this rejection; add a forwarding/password-change test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) Enforce auth for group commit stream load [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69516:
URL: https://github.com/apache/doris/pull/69516#issuecomment-6366881609
# FE Regression Coverage Report
Increment line coverage ` 46.67% (7/15)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69516_92cccd54cc8976bdf50a6d3af1403c1b35a9ec29_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69516_92cccd54cc8976bdf50a6d3af1403c1b35a9ec29_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org