You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "raghav-reglobe (via GitHub)" <gi...@apache.org> on 2026/10/19 08:36:07 UTC

[PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]

raghav-reglobe opened a new pull request, #69273:
URL: https://github.com/apache/doris/pull/69273

   ### What problem does this PR solve?
   
   Issue Number: N/A
   
   Problem Summary:
   
   A statement that must run on the master FE (`INSERT INTO ... SELECT`, `CREATE TABLE ... AS SELECT`, ...) and arrives at a FOLLOWER or OBSERVER is forwarded through `MasterOpExecutor`. The master plans it in a fresh `ConnectContext` seeded from its own global variables plus the session variables that `SessionVariable.getForwardVariables()` sends, i.e. only the fields whose `@VarAttr` carries `needForward = true` (or one of the `affectQueryResult*` attributes).
   
   `disable_join_reorder` and 37 other planner/executor knobs carried no `needForward` flag, so a session-level `SET` of any of them was silently dropped on the forwarded path:
   
   - the master could pick a different join order and distribution than the same statement run on the master FE directly, and a query that fits under its memory limit on the master fails with `MEM_LIMIT_EXCEEDED` when it enters through a follower;
   - `EXPLAIN` on the follower shows the plan the follower would run, not the one the master executes, so the difference is hard to diagnose.
   
   Reproduce on any cluster with at least two FEs: connect to a non-master FE, `SET disable_join_reorder = true;`, run an `INSERT INTO t SELECT ... FROM a JOIN b JOIN c ...` whose written join order differs from the optimizer's choice, and read the query profile: the executed plan was reordered. The same session on the master FE honours the setting.
   
   Fix: flag the 38 knobs `needForward = true`, by family:
   
   - join order: `disable_join_reorder`, `max_join_number_of_reorder`, `max_join_number_bushy_tree`, `enable_cost_based_join_reorder`, `enable_dphyp_optimizer`, `dphyper_limit`
   - distribution: `enable_bucket_shuffle_join`, `disable_colocate_plan`, `broadcast_right_table_scale_factor`, `enable_broadcast_join_force_passthrough`
   - CTE: `enable_cte_materialize`, `cte_inline_mode`, `inline_cte_referenced_threshold`
   - TopN: `enable_partition_topn`, `global_partition_topn_threshold`, `topn_opt_limit_threshold`, `topn_filter_ratio`
   - aggregation: `disable_streaming_preaggregations`, `enable_distinct_streaming_aggregation`, `enable_push_down_no_group_agg`, `agg_phase`
   - exchange: `enable_local_exchange`, `enable_local_exchange_before_agg`, `parallel_exchange_instance_num`, `exchange_multi_blocks_byte_size`, `enable_shared_exchange_sink_buffer`, `local_exchange_free_blocks_limit`
   - runtime filter: `enable_decoupled_runtime_filter`, `expand_runtime_filter_by_inner_join`, `ignore_runtime_filter_ids`
   - spill / memory: `spill_min_revocable_mem`, `spill_aggregation_partition_count`, `spill_hash_join_partition_count`, `low_memory_mode_buffer_limit`
   - skew join: `skew_rewrite_join_salt_explode_factor`
   - parallelism: `parallel_prepare_threshold`, `enable_parallel_outfile`
   - cache: `enable_sql_cache`
   
   `agg_shuffle_use_parent_key` already carries an explicit `needForward = false` and is left as it is.
   
   `SessionVariablesTest.testForwardPlanAndExecutionKnobs` checks that every knob above is present in `getForwardVariables()` and that `disable_join_reorder` round-trips through `setForwardedSessionVariables`.
   
   ### Release note
   
   Session variables that change how a statement is planned or executed (`disable_join_reorder`, the join-reorder limits, DPhyp, broadcast / bucket-shuffle / colocate switches, CTE inlining, TopN thresholds, streaming and distinct aggregation, local exchange, runtime-filter switches, spill partition counts, the salt-join explode factor, `enable_sql_cache`, ...) are now forwarded to the master FE when a statement is forwarded from a follower or observer. Previously a session-level `SET` of one of them was ignored on that path.
   
   ### Check List (For Author)
   
   - Test
       - [ ] Regression test
       - [x] Unit Test
       - [x] Manual test (add detailed scripts or steps below)
           - On a cluster with two FEs, `SET disable_join_reorder = true` on the non-master FE and run an `INSERT INTO ... SELECT` with a three-way join. Before the fix the master's query profile shows the reordered plan; after the fix it shows the written join order, identical to running the statement on the master FE.
       - [ ] No need to test or manual test. Explain why:
           - [ ] This is a refactor/code format and no logic has been changed.
           - [ ] Previous test can cover this change.
           - [ ] No code files have been changed.
           - [ ] Other reason
   
   - Behavior changed:
       - [ ] No.
       - [x] Yes. A statement forwarded to the master now honours the session variables above as if it had been run on the master directly.
   
   - Does this need documentation?
       - [x] No.
       - [ ] Yes.
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69273:
URL: https://github.com/apache/doris/pull/69273#issuecomment-6269880428

   # FE Regression Coverage Report
   Increment line coverage `  100.00% (38/38)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/69273_14fd4c630f4ec2d886cc01a859e9cab4db261351_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/69273_14fd4c630f4ec2d886cc01a859e9cab4db261351_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69273:
URL: https://github.com/apache/doris/pull/69273#issuecomment-6268957897

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 150603 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 14fd4c630f4ec2d886cc01a859e9cab4db261351, data reload: false
   
   query5	4311	580	448	448
   query6	419	212	194	194
   query7	4799	452	229	229
   query8	326	176	158	158
   query9	8765	3151	3140	3140
   query10	483	302	266	266
   query11	6048	3858	3565	3565
   query12	145	97	86	86
   query13	1262	453	318	318
   query14	6094	3955	3608	3608
   query14_1	3437	3399	3489	3399
   query15	208	204	180	180
   query16	947	427	402	402
   query17	876	694	537	537
   query18	2439	433	307	307
   query19	203	168	138	138
   query20	88	85	86	85
   query21	212	132	105	105
   query22	13006	12968	12840	12840
   query23	13302	12841	12389	12389
   query23_1	12449	12480	12604	12480
   query24	7398	843	463	463
   query24_1	476	475	490	475
   query25	535	423	365	365
   query26	1256	258	141	141
   query27	2766	450	276	276
   query28	4613	1934	1951	1934
   query29	1564	631	486	486
   query30	299	222	186	186
   query31	1030	931	827	827
   query32	142	95	93	93
   query33	536	313	246	246
   query34	904	857	500	500
   query35	761	756	650	650
   query36	798	800	700	700
   query37	132	97	87	87
   query38	1784	1742	1656	1656
   query39	798	807	742	742
   query39_1	734	728	737	728
   query40	220	138	97	97
   query41	71	70	74	70
   query42	89	87	85	85
   query43	351	379	317	317
   query44	1322	693	717	693
   query45	177	177	164	164
   query46	857	971	575	575
   query47	2977	2945	2677	2677
   query48	300	319	224	224
   query49	585	415	345	345
   query50	697	270	205	205
   query51	10474	10189	10351	10189
   query52	78	79	68	68
   query53	197	206	147	147
   query54	248	193	171	171
   query55	73	67	68	67
   query56	242	210	195	195
   query57	1637	1628	1574	1574
   query58	283	257	251	251
   query59	2069	2155	1959	1959
   query60	278	243	231	231
   query61	146	136	145	136
   query62	388	330	276	276
   query63	190	158	162	158
   query64	2733	964	818	818
   query65	3326	3259	3294	3259
   query66	1793	414	313	313
   query67	20199	20038	20008	20008
   query68	2919	1014	630	630
   query69	387	279	243	243
   query70	865	787	772	772
   query71	298	232	212	212
   query72	2733	2555	2288	2288
   query73	503	526	303	303
   query74	4639	4504	4338	4338
   query75	2333	2276	1921	1921
   query76	2308	1035	621	621
   query77	360	385	292	292
   query78	9859	9722	9159	9159
   query79	1012	861	512	512
   query80	1253	421	343	343
   query81	583	318	265	265
   query82	614	131	109	109
   query83	317	196	176	176
   query84	316	129	100	100
   query85	867	411	373	373
   query86	395	241	273	241
   query87	1986	1945	1835	1835
   query88	3751	2691	2683	2683
   query89	333	287	244	244
   query90	1908	175	165	165
   query91	164	159	129	129
   query92	100	84	88	84
   query93	1061	1014	579	579
   query94	621	292	275	275
   query95	565	389	303	303
   query96	665	488	234	234
   query97	2429	2385	2304	2304
   query98	158	157	146	146
   query99	775	758	645	645
   Total cold run time: 235576 ms
   Total hot run time: 150603 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69273:
URL: https://github.com/apache/doris/pull/69273#issuecomment-6268753203

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 28529 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 14fd4c630f4ec2d886cc01a859e9cab4db261351, data reload: false
   
   ------ Round 1 ----------------------------------
   ============================================
   q1	17612	5187	5035	5035
   q2	2076	333	266	266
   q3	10231	1385	853	853
   q4	4677	472	335	335
   q5	7435	804	531	531
   q6	181	188	154	154
   q7	746	809	612	612
   q8	9266	1302	1118	1118
   q9	5424	4243	4238	4238
   q10	6823	1290	1039	1039
   q11	429	249	230	230
   q12	643	407	298	298
   q13	18132	3559	2978	2978
   q14	282	271	248	248
   q15	q16	711	705	664	664
   q17	1187	751	581	581
   q18	7036	6134	6087	6087
   q19	1113	954	626	626
   q20	375	335	225	225
   q21	5081	2160	2353	2160
   q22	352	298	251	251
   Total cold run time: 99812 ms
   Total hot run time: 28529 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   ============================================
   q1	5604	5600	5626	5600
   q2	670	583	553	553
   q3	5202	5379	4909	4909
   q4	2030	2140	1548	1548
   q5	5247	5138	5579	5138
   q6	266	221	165	165
   q7	2348	1989	1929	1929
   q8	3038	2658	2683	2658
   q9	7974	7928	7915	7915
   q10	3768	3700	3227	3227
   q11	608	438	569	438
   q12	741	762	533	533
   q13	3531	3984	3369	3369
   q14	314	329	293	293
   q15	q16	703	703	638	638
   q17	8336	7400	6860	6860
   q18	13019	12134	12803	12134
   q19	917	810	816	810
   q20	2154	2152	1927	1927
   q21	5615	4760	4843	4760
   q22	482	509	398	398
   Total cold run time: 72567 ms
   Total hot run time: 65802 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69273:
URL: https://github.com/apache/doris/pull/69273#issuecomment-6263132139

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #69273:
URL: https://github.com/apache/doris/pull/69273#issuecomment-6263624865

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #69273:
URL: https://github.com/apache/doris/pull/69273#issuecomment-6263626961

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69273:
URL: https://github.com/apache/doris/pull/69273#issuecomment-6269056241

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 24.42 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 14fd4c630f4ec2d886cc01a859e9cab4db261351, data reload: false
   
   query1	0.01	0.01	0.00
   query2	0.09	0.05	0.05
   query3	0.26	0.09	0.09
   query4	1.62	0.10	0.11
   query5	0.22	0.25	0.24
   query6	1.17	0.68	0.67
   query7	0.04	0.01	0.00
   query8	0.05	0.04	0.04
   query9	0.41	0.35	0.35
   query10	0.59	0.59	0.59
   query11	0.20	0.15	0.15
   query12	0.17	0.14	0.14
   query13	0.50	0.51	0.50
   query14	1.02	1.02	1.01
   query15	0.62	0.60	0.61
   query16	0.36	0.36	0.35
   query17	1.16	1.13	1.16
   query18	0.21	0.20	0.18
   query19	2.05	1.94	2.00
   query20	0.02	0.02	0.01
   query21	15.48	0.20	0.13
   query22	4.91	0.05	0.06
   query23	16.29	0.32	0.13
   query24	2.65	0.57	0.39
   query25	0.13	0.07	0.06
   query26	1.23	0.20	0.15
   query27	0.05	0.05	0.05
   query28	3.45	0.74	0.37
   query29	12.48	4.14	3.33
   query30	0.27	0.12	0.09
   query31	2.76	0.56	0.32
   query32	3.23	0.60	0.50
   query33	3.18	3.29	3.23
   query34	15.30	4.00	3.43
   query35	3.38	3.35	3.31
   query36	0.54	0.43	0.41
   query37	0.09	0.07	0.06
   query38	0.04	0.04	0.03
   query39	0.03	0.02	0.02
   query40	0.17	0.15	0.14
   query41	0.09	0.03	0.03
   query42	0.03	0.02	0.02
   query43	0.04	0.03	0.03
   Total cold run time: 96.59 s
   Total hot run time: 24.42 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](nereids) forward plan/execution-affecting session variables to the master [doris]

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 merged PR #69273:
URL: https://github.com/apache/doris/pull/69273


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org