You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "raghav-reglobe (via GitHub)" <gi...@apache.org> on 2026/10/19 08:36:07 UTC
[PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]
raghav-reglobe opened a new pull request, #69273:
URL: https://github.com/apache/doris/pull/69273
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
A statement that must run on the master FE (`INSERT INTO ... SELECT`, `CREATE TABLE ... AS SELECT`, ...) and arrives at a FOLLOWER or OBSERVER is forwarded through `MasterOpExecutor`. The master plans it in a fresh `ConnectContext` seeded from its own global variables plus the session variables that `SessionVariable.getForwardVariables()` sends, i.e. only the fields whose `@VarAttr` carries `needForward = true` (or one of the `affectQueryResult*` attributes).
`disable_join_reorder` and 37 other planner/executor knobs carried no `needForward` flag, so a session-level `SET` of any of them was silently dropped on the forwarded path:
- the master could pick a different join order and distribution than the same statement run on the master FE directly, and a query that fits under its memory limit on the master fails with `MEM_LIMIT_EXCEEDED` when it enters through a follower;
- `EXPLAIN` on the follower shows the plan the follower would run, not the one the master executes, so the difference is hard to diagnose.
Reproduce on any cluster with at least two FEs: connect to a non-master FE, `SET disable_join_reorder = true;`, run an `INSERT INTO t SELECT ... FROM a JOIN b JOIN c ...` whose written join order differs from the optimizer's choice, and read the query profile: the executed plan was reordered. The same session on the master FE honours the setting.
Fix: flag the 38 knobs `needForward = true`, by family:
- join order: `disable_join_reorder`, `max_join_number_of_reorder`, `max_join_number_bushy_tree`, `enable_cost_based_join_reorder`, `enable_dphyp_optimizer`, `dphyper_limit`
- distribution: `enable_bucket_shuffle_join`, `disable_colocate_plan`, `broadcast_right_table_scale_factor`, `enable_broadcast_join_force_passthrough`
- CTE: `enable_cte_materialize`, `cte_inline_mode`, `inline_cte_referenced_threshold`
- TopN: `enable_partition_topn`, `global_partition_topn_threshold`, `topn_opt_limit_threshold`, `topn_filter_ratio`
- aggregation: `disable_streaming_preaggregations`, `enable_distinct_streaming_aggregation`, `enable_push_down_no_group_agg`, `agg_phase`
- exchange: `enable_local_exchange`, `enable_local_exchange_before_agg`, `parallel_exchange_instance_num`, `exchange_multi_blocks_byte_size`, `enable_shared_exchange_sink_buffer`, `local_exchange_free_blocks_limit`
- runtime filter: `enable_decoupled_runtime_filter`, `expand_runtime_filter_by_inner_join`, `ignore_runtime_filter_ids`
- spill / memory: `spill_min_revocable_mem`, `spill_aggregation_partition_count`, `spill_hash_join_partition_count`, `low_memory_mode_buffer_limit`
- skew join: `skew_rewrite_join_salt_explode_factor`
- parallelism: `parallel_prepare_threshold`, `enable_parallel_outfile`
- cache: `enable_sql_cache`
`agg_shuffle_use_parent_key` already carries an explicit `needForward = false` and is left as it is.
`SessionVariablesTest.testForwardPlanAndExecutionKnobs` checks that every knob above is present in `getForwardVariables()` and that `disable_join_reorder` round-trips through `setForwardedSessionVariables`.
### Release note
Session variables that change how a statement is planned or executed (`disable_join_reorder`, the join-reorder limits, DPhyp, broadcast / bucket-shuffle / colocate switches, CTE inlining, TopN thresholds, streaming and distinct aggregation, local exchange, runtime-filter switches, spill partition counts, the salt-join explode factor, `enable_sql_cache`, ...) are now forwarded to the master FE when a statement is forwarded from a follower or observer. Previously a session-level `SET` of one of them was ignored on that path.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- On a cluster with two FEs, `SET disable_join_reorder = true` on the non-master FE and run an `INSERT INTO ... SELECT` with a three-way join. Before the fix the master's query profile shows the reordered plan; after the fix it shows the written join order, identical to running the statement on the master FE.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. A statement forwarded to the master now honours the session variables above as if it had been run on the master directly.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69273:
URL: https://github.com/apache/doris/pull/69273#issuecomment-6269880428
# FE Regression Coverage Report
Increment line coverage ` 100.00% (38/38)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69273_14fd4c630f4ec2d886cc01a859e9cab4db261351_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69273_14fd4c630f4ec2d886cc01a859e9cab4db261351_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69273:
URL: https://github.com/apache/doris/pull/69273#issuecomment-6268957897
<details>
<summary>TPC-DS: <b>Total hot run time: 150603 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 14fd4c630f4ec2d886cc01a859e9cab4db261351, data reload: false
query5 4311 580 448 448
query6 419 212 194 194
query7 4799 452 229 229
query8 326 176 158 158
query9 8765 3151 3140 3140
query10 483 302 266 266
query11 6048 3858 3565 3565
query12 145 97 86 86
query13 1262 453 318 318
query14 6094 3955 3608 3608
query14_1 3437 3399 3489 3399
query15 208 204 180 180
query16 947 427 402 402
query17 876 694 537 537
query18 2439 433 307 307
query19 203 168 138 138
query20 88 85 86 85
query21 212 132 105 105
query22 13006 12968 12840 12840
query23 13302 12841 12389 12389
query23_1 12449 12480 12604 12480
query24 7398 843 463 463
query24_1 476 475 490 475
query25 535 423 365 365
query26 1256 258 141 141
query27 2766 450 276 276
query28 4613 1934 1951 1934
query29 1564 631 486 486
query30 299 222 186 186
query31 1030 931 827 827
query32 142 95 93 93
query33 536 313 246 246
query34 904 857 500 500
query35 761 756 650 650
query36 798 800 700 700
query37 132 97 87 87
query38 1784 1742 1656 1656
query39 798 807 742 742
query39_1 734 728 737 728
query40 220 138 97 97
query41 71 70 74 70
query42 89 87 85 85
query43 351 379 317 317
query44 1322 693 717 693
query45 177 177 164 164
query46 857 971 575 575
query47 2977 2945 2677 2677
query48 300 319 224 224
query49 585 415 345 345
query50 697 270 205 205
query51 10474 10189 10351 10189
query52 78 79 68 68
query53 197 206 147 147
query54 248 193 171 171
query55 73 67 68 67
query56 242 210 195 195
query57 1637 1628 1574 1574
query58 283 257 251 251
query59 2069 2155 1959 1959
query60 278 243 231 231
query61 146 136 145 136
query62 388 330 276 276
query63 190 158 162 158
query64 2733 964 818 818
query65 3326 3259 3294 3259
query66 1793 414 313 313
query67 20199 20038 20008 20008
query68 2919 1014 630 630
query69 387 279 243 243
query70 865 787 772 772
query71 298 232 212 212
query72 2733 2555 2288 2288
query73 503 526 303 303
query74 4639 4504 4338 4338
query75 2333 2276 1921 1921
query76 2308 1035 621 621
query77 360 385 292 292
query78 9859 9722 9159 9159
query79 1012 861 512 512
query80 1253 421 343 343
query81 583 318 265 265
query82 614 131 109 109
query83 317 196 176 176
query84 316 129 100 100
query85 867 411 373 373
query86 395 241 273 241
query87 1986 1945 1835 1835
query88 3751 2691 2683 2683
query89 333 287 244 244
query90 1908 175 165 165
query91 164 159 129 129
query92 100 84 88 84
query93 1061 1014 579 579
query94 621 292 275 275
query95 565 389 303 303
query96 665 488 234 234
query97 2429 2385 2304 2304
query98 158 157 146 146
query99 775 758 645 645
Total cold run time: 235576 ms
Total hot run time: 150603 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69273:
URL: https://github.com/apache/doris/pull/69273#issuecomment-6268753203
<details>
<summary>TPC-H: <b>Total hot run time: 28529 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 14fd4c630f4ec2d886cc01a859e9cab4db261351, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17612 5187 5035 5035
q2 2076 333 266 266
q3 10231 1385 853 853
q4 4677 472 335 335
q5 7435 804 531 531
q6 181 188 154 154
q7 746 809 612 612
q8 9266 1302 1118 1118
q9 5424 4243 4238 4238
q10 6823 1290 1039 1039
q11 429 249 230 230
q12 643 407 298 298
q13 18132 3559 2978 2978
q14 282 271 248 248
q15 q16 711 705 664 664
q17 1187 751 581 581
q18 7036 6134 6087 6087
q19 1113 954 626 626
q20 375 335 225 225
q21 5081 2160 2353 2160
q22 352 298 251 251
Total cold run time: 99812 ms
Total hot run time: 28529 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5604 5600 5626 5600
q2 670 583 553 553
q3 5202 5379 4909 4909
q4 2030 2140 1548 1548
q5 5247 5138 5579 5138
q6 266 221 165 165
q7 2348 1989 1929 1929
q8 3038 2658 2683 2658
q9 7974 7928 7915 7915
q10 3768 3700 3227 3227
q11 608 438 569 438
q12 741 762 533 533
q13 3531 3984 3369 3369
q14 314 329 293 293
q15 q16 703 703 638 638
q17 8336 7400 6860 6860
q18 13019 12134 12803 12134
q19 917 810 816 810
q20 2154 2152 1927 1927
q21 5615 4760 4843 4760
q22 482 509 398 398
Total cold run time: 72567 ms
Total hot run time: 65802 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69273:
URL: https://github.com/apache/doris/pull/69273#issuecomment-6263132139
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #69273:
URL: https://github.com/apache/doris/pull/69273#issuecomment-6263624865
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #69273:
URL: https://github.com/apache/doris/pull/69273#issuecomment-6263626961
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69273:
URL: https://github.com/apache/doris/pull/69273#issuecomment-6269056241
<details>
<summary>ClickBench: <b>Total hot run time: 24.42 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 14fd4c630f4ec2d886cc01a859e9cab4db261351, data reload: false
query1 0.01 0.01 0.00
query2 0.09 0.05 0.05
query3 0.26 0.09 0.09
query4 1.62 0.10 0.11
query5 0.22 0.25 0.24
query6 1.17 0.68 0.67
query7 0.04 0.01 0.00
query8 0.05 0.04 0.04
query9 0.41 0.35 0.35
query10 0.59 0.59 0.59
query11 0.20 0.15 0.15
query12 0.17 0.14 0.14
query13 0.50 0.51 0.50
query14 1.02 1.02 1.01
query15 0.62 0.60 0.61
query16 0.36 0.36 0.35
query17 1.16 1.13 1.16
query18 0.21 0.20 0.18
query19 2.05 1.94 2.00
query20 0.02 0.02 0.01
query21 15.48 0.20 0.13
query22 4.91 0.05 0.06
query23 16.29 0.32 0.13
query24 2.65 0.57 0.39
query25 0.13 0.07 0.06
query26 1.23 0.20 0.15
query27 0.05 0.05 0.05
query28 3.45 0.74 0.37
query29 12.48 4.14 3.33
query30 0.27 0.12 0.09
query31 2.76 0.56 0.32
query32 3.23 0.60 0.50
query33 3.18 3.29 3.23
query34 15.30 4.00 3.43
query35 3.38 3.35 3.31
query36 0.54 0.43 0.41
query37 0.09 0.07 0.06
query38 0.04 0.04 0.03
query39 0.03 0.02 0.02
query40 0.17 0.15 0.14
query41 0.09 0.03 0.03
query42 0.03 0.02 0.02
query43 0.04 0.03 0.03
Total cold run time: 96.59 s
Total hot run time: 24.42 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 merged PR #69273:
URL: https://github.com/apache/doris/pull/69273
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org