You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "2010YOUY01 (via GitHub)" <gi...@apache.org> on 2025/09/08 06:03:54 UTC
[PR] feat: Make join selection configurable through `enable_*` and `prefer_*` options [datafusion]
2010YOUY01 opened a new pull request, #17467:
URL: https://github.com/apache/datafusion/pull/17467
…tions
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Closes https://github.com/apache/datafusion/issues/17432
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
See the issue for details.
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
1. Add configuration options to make join type selection configurable.
2. Move the join type selection logic from `physical_planner.rs` into `join_planner.rs`, and update it to follow flags added in step 1. No existing planning logic has changed.
3. Tested different flag combinations through a `slt`
4. Fix existing tests due to `datafusion.optimizer.prefer_hash_join`'s semantics change. See below breaking change section and the issue writeup for details.
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
Yes, see above.
## Are there any user-facing changes?
Yes, there is a semantics change for a existing configuration option:
```
datafusion.optimizer.prefer_hash_join | true | When set to true, the physical plan optimizer will prefer HashJoin over SortMergeJoin. HashJoin can work more efficiently than SortMergeJoin but consumes more memory
```
The previous semantics of this flag is toggling between HJ and SMJ: if true use HJ, if false use SMJ. Now if we want to prefer SMJ, we should set `datafusion.optimizer.prefer_sort_merge_join` to true.
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Make join selection configurable through `enable_*` and `join_method_priority` options [datafusion]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on PR #17467:
URL: https://github.com/apache/datafusion/pull/17467#issuecomment-3299798128
@alamb Do you know if it will include join ordering improvements?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Make join selection configurable through `enable_*` and `prefer_*` options [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on PR #17467:
URL: https://github.com/apache/datafusion/pull/17467#issuecomment-3271508741
@alamb Thanks for the suggestions:
- Regarding the ambiguous `prefer_*` setting, WDYT using a single config instead: `join_method_priority = 'hj, smj, nlj'` instead?
- I'll think more about this `JoinPlanner` idea later. I think making join ordering configurable is important too. What I’m planning to do next is to add a configuration option to disable the built-in join reordering, as the simplest join order option . Let me think about how to incorporate that into JoinPlanner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Make join selection configurable through `enable_*` and `join_method_priority` options [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on code in PR #17467:
URL: https://github.com/apache/datafusion/pull/17467#discussion_r2347196158
##########
datafusion/core/src/physical_planner/join_planner.rs:
##########
@@ -0,0 +1,212 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use crate::error::Result;
+use crate::execution::context::SessionState;
+use crate::physical_plan::joins::utils as join_utils;
+use crate::physical_plan::joins::{
+ CrossJoinExec, HashJoinExec, NestedLoopJoinExec, PartitionMode, SortMergeJoinExec,
+};
+use crate::physical_plan::ExecutionPlan;
+use arrow::compute::SortOptions;
+use datafusion_common::{config_err, plan_err};
+use datafusion_expr::JoinType;
+
+/// Build the appropriate join `ExecutionPlan` for the given join type, filter, and
+/// configurations.
+///
+/// For example, given an equi-join, the planner may execute it as a Nested Loop
+/// Join, Hash Join, or another strategy. Configuration settings determine which
+/// ExecutionPlan is used.
+///
+/// # Strategy
+/// - Step 1: Find all possible physical join types for the given join logical plan
+/// - No join on keys and no filter => CrossJoin
+/// - With equality? => HJ and SMJ (if with multiple partition)
+/// TODO: The constraint on SMJ is added previously for optimization. Should
+/// we remove it for configurability?
+/// TODO: Allow NLJ for equal join for better configurability.
+/// - Without equality? => NLJ
+/// - Step 2: Filter the possible join types from step 1 according to the configuration
+/// by checking if they're enabled by options like `datafusion.optimizer.enable_hash_join`
+/// - Step 3: Choose one according to the built-in heuristics and also the preference
+/// in the configuration `datafusion.optimizer.join_method_priority`
+pub(super) fn plan_initial_join_exec(
+ session_state: &SessionState,
+ physical_left: Arc<dyn ExecutionPlan>,
+ physical_right: Arc<dyn ExecutionPlan>,
+ join_on: join_utils::JoinOn,
+ join_filter: Option<join_utils::JoinFilter>,
+ join_type: &JoinType,
+ null_equality: &datafusion_common::NullEquality,
+) -> Result<Arc<dyn ExecutionPlan>> {
+ // Short-circuit: handle pure cross join (existing behavior)
+ if join_on.is_empty() && join_filter.is_none() && matches!(join_type, JoinType::Inner)
+ {
+ return Ok(Arc::new(CrossJoinExec::new(physical_left, physical_right)));
+ }
+
+ // Step 1: Find possible join types for the given Logical Plan
+ // ----------------------------------------------------------------------
+
+ // Build the list of possible algorithms for this join
+ #[derive(Debug, Clone, Copy, PartialEq, Eq)]
+ enum Algo {
+ Nlj,
+ Hj,
+ Smj,
+ }
+
+ let cfg = &session_state.config_options().optimizer;
+ let can_smj = session_state.config().target_partitions() > 1
+ && session_state.config().repartition_joins();
+
+ let mut possible: Vec<Algo> = Vec::new();
+ if join_on.is_empty() {
+ possible.push(Algo::Nlj);
+ } else {
+ possible.push(Algo::Hj);
+ if can_smj {
+ possible.push(Algo::Smj);
+ }
+ }
+
+ // Step 2: Filter the possible list according to enable flags from config
+ // ----------------------------------------------------------------------
+
+ // Filter by enable flags
+ let enabled_and_possible: Vec<Algo> = possible
+ .iter()
+ .copied()
+ .filter(|a| match a {
+ Algo::Nlj => cfg.enable_nested_loop_join,
+ Algo::Hj => cfg.enable_hash_join,
+ Algo::Smj => cfg.enable_sort_merge_join,
+ })
+ .collect();
+
+ if enabled_and_possible.is_empty() {
+ return plan_err!(
+ "No enabled join algorithm is applicable for this join. Possible join types are {:?}. Try to enable them through configurations like `datafusion.optimizer.enable_hash_join`", &possible
+ );
+ }
+
+ // Step 3: Choose and plan the physical join type according to
+ // `join_method_priority` and built-in heuristics
+ // ----------------------------------------------------------------------
+
+ // Parse join method priority string into an ordered list of algorithms
+ let parse_priority = |s: &str| -> Result<Vec<Algo>> {
+ let mut out = Vec::new();
+ let mut unknown = Vec::new();
+ for token in s.split(',').map(|t| t.trim()).filter(|t| !t.is_empty()) {
+ match token {
+ "hj" | "hash_join" => out.push(Algo::Hj),
+ "smj" | "sort_merge_join" => out.push(Algo::Smj),
+ "nlj" | "nested_loop_join" => out.push(Algo::Nlj),
+ other => unknown.push(other.to_string()),
+ }
+ }
+ if !unknown.is_empty() {
+ let valid = "hj/hash_join, smj/sort_merge_join, nlj/nested_loop_join";
+ return config_err!(
+ "Invalid join method(s) in datafusion.optimizer.join_method_priority: {}. Valid values: {}",
+ unknown.join(", "),
+ valid
+ );
+ }
+ Ok(out)
+ };
+
+ // Backward compatibility:
+ // If `join_method_priority` is empty, honor legacy `prefer_hash_join` by
+ // setting the priority to a single entry accordingly. Otherwise, parse the
Review Comment:
It will be noticeably slow for very large joins (e.g. joining 1000 table), but I think by then planning speed would be the last thing to worry about.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Make join selection configurable through `enable_*` and `prefer_*` options [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #17467:
URL: https://github.com/apache/datafusion/pull/17467#discussion_r2333593307
##########
datafusion/core/src/physical_planner/join_planner.rs:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use crate::error::Result;
+use crate::execution::context::SessionState;
+use crate::physical_plan::joins::utils as join_utils;
+use crate::physical_plan::joins::{
+ CrossJoinExec, HashJoinExec, NestedLoopJoinExec, PartitionMode, SortMergeJoinExec,
+};
+use crate::physical_plan::ExecutionPlan;
+use arrow::compute::SortOptions;
+use datafusion_common::plan_err;
+use datafusion_expr::JoinType;
+
+/// Build the appropriate join `ExecutionPlan` for the given join type, filter, and
+/// configurations.
+///
+/// For example, given an equi-join, the planner may execute it as a Nested Loop
+/// Join, Hash Join, or another strategy. Configuration settings determine which
+/// ExecutionPlan is used.
+///
+/// # Strategy
+/// - Step 1: Find all possible physical join types for the given join logical plan
+/// - No join on keys and no filter => CrossJoin
+/// - With equality? => HJ and SMJ (if with multiple partition)
+/// TODO: The constraint on SMJ is added previously for optimization. Should
+/// we remove it for configurability?
+/// TODO: Allow NLJ for equal join for better configurability.
+/// - Without equality? => NLJ
+/// - Step 2: Filter the possible join types from step 1 according to the configuration
+/// by checking if they're enabled by options like `datafusion.optimizer.enable_hash_join`
+/// - Step 3: Choose one according to the built-in heuristics and also the preference
+/// in the configuration, e.g. `datafusion.optimizer.prefer_hash_join`
+pub(super) fn plan_join_exec(
Review Comment:
Maybe this could be called "plan_initial_join_exec"
##########
datafusion/core/src/physical_planner/join_planner.rs:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use crate::error::Result;
+use crate::execution::context::SessionState;
+use crate::physical_plan::joins::utils as join_utils;
+use crate::physical_plan::joins::{
+ CrossJoinExec, HashJoinExec, NestedLoopJoinExec, PartitionMode, SortMergeJoinExec,
+};
+use crate::physical_plan::ExecutionPlan;
+use arrow::compute::SortOptions;
+use datafusion_common::plan_err;
+use datafusion_expr::JoinType;
+
+/// Build the appropriate join `ExecutionPlan` for the given join type, filter, and
+/// configurations.
+///
+/// For example, given an equi-join, the planner may execute it as a Nested Loop
+/// Join, Hash Join, or another strategy. Configuration settings determine which
+/// ExecutionPlan is used.
+///
+/// # Strategy
+/// - Step 1: Find all possible physical join types for the given join logical plan
+/// - No join on keys and no filter => CrossJoin
+/// - With equality? => HJ and SMJ (if with multiple partition)
+/// TODO: The constraint on SMJ is added previously for optimization. Should
+/// we remove it for configurability?
+/// TODO: Allow NLJ for equal join for better configurability.
+/// - Without equality? => NLJ
+/// - Step 2: Filter the possible join types from step 1 according to the configuration
+/// by checking if they're enabled by options like `datafusion.optimizer.enable_hash_join`
+/// - Step 3: Choose one according to the built-in heuristics and also the preference
+/// in the configuration, e.g. `datafusion.optimizer.prefer_hash_join`
+pub(super) fn plan_join_exec(
+ session_state: &SessionState,
+ physical_left: Arc<dyn ExecutionPlan>,
+ physical_right: Arc<dyn ExecutionPlan>,
+ join_on: join_utils::JoinOn,
+ join_filter: Option<join_utils::JoinFilter>,
+ join_type: &JoinType,
+ null_equality: &datafusion_common::NullEquality,
+) -> Result<Arc<dyn ExecutionPlan>> {
+ // Short-circuit: handle pure cross join (existing behavior)
+ if join_on.is_empty() && join_filter.is_none() && matches!(join_type, JoinType::Inner)
+ {
+ return Ok(Arc::new(CrossJoinExec::new(physical_left, physical_right)));
+ }
+
+ // Step 1: Find possible join types for the given Logical Plan
+ // ----------------------------------------------------------------------
+
+ // Build the list of possible algorithms for this join
+ #[derive(Debug, Clone, Copy, PartialEq, Eq)]
+ enum Algo {
+ Nlj,
+ Hj,
+ Smj,
+ }
+
+ let cfg = &session_state.config_options().optimizer;
+ let can_smj = session_state.config().target_partitions() > 1
+ && session_state.config().repartition_joins();
+
+ let mut possible: Vec<Algo> = Vec::new();
+ if join_on.is_empty() {
+ possible.push(Algo::Nlj);
+ } else {
+ possible.push(Algo::Hj);
+ if can_smj {
+ possible.push(Algo::Smj);
+ }
+ }
+
+ // Step 2: Filter the possible list according to enable flags from config
+ // ----------------------------------------------------------------------
+
+ // Filter by enable flags
+ let enabled_and_possible: Vec<Algo> = possible
+ .iter()
+ .copied()
+ .filter(|a| match a {
+ Algo::Nlj => cfg.enable_nested_loop_join,
+ Algo::Hj => cfg.enable_hash_join,
+ Algo::Smj => cfg.enable_sort_merge_join,
+ })
+ .collect();
+
+ if enabled_and_possible.is_empty() {
+ return plan_err!(
+ "No enabled join algorithm is applicable for this join. Possible join types are {:?}. Try to enable them through configurations like `datafusion.optimizer.enable_hash_join`", &possible
+ );
+ }
+
+ // Step 3: Choose and plan the physical join type according to preference
+ // from the configuration, and also the built-in heuristics
+ // ----------------------------------------------------------------------
+
+ // Collect preferred algorithms
+ let mut preferred: Vec<Algo> = Vec::new();
+ if cfg.prefer_hash_join {
+ preferred.push(Algo::Hj);
+ }
+ if cfg.prefer_sort_merge_join {
+ preferred.push(Algo::Smj);
+ }
+ if cfg.prefer_nested_loop_join {
+ preferred.push(Algo::Nlj);
+ }
+
+ // Helper to pick by priority HJ > SMJ > NLJ
Review Comment:
this is an important detail (the relative priorities) of the different join algorithms that we should probably communicate to the user somehow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Make join selection configurable through `enable_*` and `join_method_priority` options [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #17467: feat: Make join selection configurable through `enable_*` and `join_method_priority` options
URL: https://github.com/apache/datafusion/pull/17467
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Make join selection configurable through `enable_*` and `join_method_priority` options [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on PR #17467:
URL: https://github.com/apache/datafusion/pull/17467#issuecomment-3288530256
> Given the increasing interest in improving Joins in DataFusion, I wonder if now is the time to create some space / a structure for more sophisticated join planners instead of making the existing one more complicated. In particular, I think the join algorithm is just one part of a more sophisticated strategy for joins (that also may reorders joins, for example)
>
> Maybe we could make `JoinPlanner` a trait that can be registered with the SessionContext or the Optimizer the same way as ExtensionPLanners?
>
> Then we can provide a default JoinPlanner (what currently exists) that has its own config namespace, etc
>
> ```rust
> trait JoinPlanner {
> // plan the initial join when converting from Logical --> Physical join
> fn plan_initial_join(
> session_state: &SessionState,
> physical_left: Arc<dyn ExecutionPlan>,
> physical_right: Arc<dyn ExecutionPlan>,
> join_on: join_utils::JoinOn,
> join_filter: Option<join_utils::JoinFilter>,
> join_type: &JoinType,
> null_equality: &datafusion_common::NullEquality,) -> Arc<dyn ExecutionPlan>;
> }
> ```
I think this trait should include two major steps from the current implementation:
1. `plan_initia_join` method: it converts the join logical plan to initial physical plan, which decides which join method to use (hash join or NLJ) for join nodes
2. `JoinSelection` optimizer rule: it further refine the initial physical plan (e.g. change the repartition strategy inside Hash Join's initial physical plan), and also swaps join inputs according to stats
Now I think it's a bit hard to extract them into a pluggable module, because they seem tightly coupled with other optimizer rules. Perhaps we can give it a try when there are multiple radically different join planning/reordering strategies available — we'll have a better understanding of how this interface should look by then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Make join selection configurable through `enable_*` and `join_method_priority` options [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #17467:
URL: https://github.com/apache/datafusion/pull/17467#issuecomment-3300277479
I think the idea is to provide some basic join ordering, but mostly focus on the APIs that would permit people to plug in more sophisticated join orderings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Make join selection configurable through `enable_*` and `prefer_*` options [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #17467:
URL: https://github.com/apache/datafusion/pull/17467#issuecomment-3276353751
> * Regarding the ambiguous `prefer_*` setting, WDYT using a single config instead: `join_method_priority = 'hj, smj, nlj'` instead?
I think that would be better than three distinct settings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Make join selection configurable through `enable_*` and `join_method_priority` options [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #17467:
URL: https://github.com/apache/datafusion/pull/17467#issuecomment-3319026562
> I think the idea is to provide some basic join ordering, but mostly focus on the APIs that would permit people to plug in more sophisticated join orderings.
I filed a ticket to discuss a potentially more general approach to join ordering and access path selection
- https://github.com/apache/datafusion/issues/17718
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Make join selection configurable through `enable_*` and `join_method_priority` options [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on code in PR #17467:
URL: https://github.com/apache/datafusion/pull/17467#discussion_r2346792135
##########
datafusion/core/src/physical_planner/join_planner.rs:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use crate::error::Result;
+use crate::execution::context::SessionState;
+use crate::physical_plan::joins::utils as join_utils;
+use crate::physical_plan::joins::{
+ CrossJoinExec, HashJoinExec, NestedLoopJoinExec, PartitionMode, SortMergeJoinExec,
+};
+use crate::physical_plan::ExecutionPlan;
+use arrow::compute::SortOptions;
+use datafusion_common::plan_err;
+use datafusion_expr::JoinType;
+
+/// Build the appropriate join `ExecutionPlan` for the given join type, filter, and
+/// configurations.
+///
+/// For example, given an equi-join, the planner may execute it as a Nested Loop
+/// Join, Hash Join, or another strategy. Configuration settings determine which
+/// ExecutionPlan is used.
+///
+/// # Strategy
+/// - Step 1: Find all possible physical join types for the given join logical plan
+/// - No join on keys and no filter => CrossJoin
+/// - With equality? => HJ and SMJ (if with multiple partition)
+/// TODO: The constraint on SMJ is added previously for optimization. Should
+/// we remove it for configurability?
+/// TODO: Allow NLJ for equal join for better configurability.
+/// - Without equality? => NLJ
+/// - Step 2: Filter the possible join types from step 1 according to the configuration
+/// by checking if they're enabled by options like `datafusion.optimizer.enable_hash_join`
+/// - Step 3: Choose one according to the built-in heuristics and also the preference
+/// in the configuration, e.g. `datafusion.optimizer.prefer_hash_join`
+pub(super) fn plan_join_exec(
+ session_state: &SessionState,
+ physical_left: Arc<dyn ExecutionPlan>,
+ physical_right: Arc<dyn ExecutionPlan>,
+ join_on: join_utils::JoinOn,
+ join_filter: Option<join_utils::JoinFilter>,
+ join_type: &JoinType,
+ null_equality: &datafusion_common::NullEquality,
+) -> Result<Arc<dyn ExecutionPlan>> {
+ // Short-circuit: handle pure cross join (existing behavior)
+ if join_on.is_empty() && join_filter.is_none() && matches!(join_type, JoinType::Inner)
+ {
+ return Ok(Arc::new(CrossJoinExec::new(physical_left, physical_right)));
+ }
+
+ // Step 1: Find possible join types for the given Logical Plan
+ // ----------------------------------------------------------------------
+
+ // Build the list of possible algorithms for this join
+ #[derive(Debug, Clone, Copy, PartialEq, Eq)]
+ enum Algo {
+ Nlj,
+ Hj,
+ Smj,
+ }
+
+ let cfg = &session_state.config_options().optimizer;
+ let can_smj = session_state.config().target_partitions() > 1
+ && session_state.config().repartition_joins();
+
+ let mut possible: Vec<Algo> = Vec::new();
+ if join_on.is_empty() {
+ possible.push(Algo::Nlj);
+ } else {
+ possible.push(Algo::Hj);
+ if can_smj {
+ possible.push(Algo::Smj);
+ }
+ }
+
+ // Step 2: Filter the possible list according to enable flags from config
+ // ----------------------------------------------------------------------
+
+ // Filter by enable flags
+ let enabled_and_possible: Vec<Algo> = possible
+ .iter()
+ .copied()
+ .filter(|a| match a {
+ Algo::Nlj => cfg.enable_nested_loop_join,
+ Algo::Hj => cfg.enable_hash_join,
+ Algo::Smj => cfg.enable_sort_merge_join,
+ })
+ .collect();
+
+ if enabled_and_possible.is_empty() {
+ return plan_err!(
+ "No enabled join algorithm is applicable for this join. Possible join types are {:?}. Try to enable them through configurations like `datafusion.optimizer.enable_hash_join`", &possible
+ );
+ }
+
+ // Step 3: Choose and plan the physical join type according to preference
+ // from the configuration, and also the built-in heuristics
+ // ----------------------------------------------------------------------
+
+ // Collect preferred algorithms
+ let mut preferred: Vec<Algo> = Vec::new();
+ if cfg.prefer_hash_join {
+ preferred.push(Algo::Hj);
+ }
+ if cfg.prefer_sort_merge_join {
+ preferred.push(Algo::Smj);
+ }
+ if cfg.prefer_nested_loop_join {
+ preferred.push(Algo::Nlj);
+ }
+
+ // Helper to pick by priority HJ > SMJ > NLJ
Review Comment:
added in the config manual
##########
datafusion/core/src/physical_planner/join_planner.rs:
##########
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use crate::error::Result;
+use crate::execution::context::SessionState;
+use crate::physical_plan::joins::utils as join_utils;
+use crate::physical_plan::joins::{
+ CrossJoinExec, HashJoinExec, NestedLoopJoinExec, PartitionMode, SortMergeJoinExec,
+};
+use crate::physical_plan::ExecutionPlan;
+use arrow::compute::SortOptions;
+use datafusion_common::plan_err;
+use datafusion_expr::JoinType;
+
+/// Build the appropriate join `ExecutionPlan` for the given join type, filter, and
+/// configurations.
+///
+/// For example, given an equi-join, the planner may execute it as a Nested Loop
+/// Join, Hash Join, or another strategy. Configuration settings determine which
+/// ExecutionPlan is used.
+///
+/// # Strategy
+/// - Step 1: Find all possible physical join types for the given join logical plan
+/// - No join on keys and no filter => CrossJoin
+/// - With equality? => HJ and SMJ (if with multiple partition)
+/// TODO: The constraint on SMJ is added previously for optimization. Should
+/// we remove it for configurability?
+/// TODO: Allow NLJ for equal join for better configurability.
+/// - Without equality? => NLJ
+/// - Step 2: Filter the possible join types from step 1 according to the configuration
+/// by checking if they're enabled by options like `datafusion.optimizer.enable_hash_join`
+/// - Step 3: Choose one according to the built-in heuristics and also the preference
+/// in the configuration, e.g. `datafusion.optimizer.prefer_hash_join`
+pub(super) fn plan_join_exec(
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Make join selection configurable through `enable_*` and `prefer_*` options [datafusion]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17467:
URL: https://github.com/apache/datafusion/pull/17467#discussion_r2329219849
##########
datafusion/core/src/physical_planner/join_planner.rs:
##########
@@ -0,0 +1,196 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use crate::error::Result;
+use crate::execution::context::SessionState;
+use crate::physical_plan::joins::utils as join_utils;
+use crate::physical_plan::joins::{
+ CrossJoinExec, HashJoinExec, NestedLoopJoinExec, PartitionMode, SortMergeJoinExec,
+};
+use crate::physical_plan::ExecutionPlan;
+use arrow::compute::SortOptions;
+use datafusion_common::plan_err;
+use datafusion_expr::JoinType;
+
+/// Build the appropriate join `ExecutionPlan` for the given join type, filter, and
+/// configurations.
+///
+/// For example, given an equi-join, the planner may execute it as a Nested Loop
+/// Join, Hash Join, or another strategy. Configuration settings determine which
+/// ExecutionPlan is used.
+///
+/// # Strategy
+/// - Step 1: Find all possible physical join types for the given join logical plan
+/// - No join on keys and no filter => CrossJoin
+/// - With equality? => HJ and SMJ(if with multiple partition)
+/// TODO: The constraint on SMJ is added previously for optimization. Should
+/// we remove it for configurability?
+/// TODO: Allow NLJ for equal join for better configurability.
+/// - Without equality? => NLJ
+/// - Step 2: Filter the possible join types from step 1 according to the configuration
+/// , by checking if they're enabled by options like `datafusion.optimizer.enable_hash_join`
+/// - Step 3: Choose one according to the built-in heuristics and also the preference
+/// in the configuration, e.g. `datafusion.optimizer.prefer_hash_join`
+pub(super) fn plan_join_exec(
+ session_state: &SessionState,
+ physical_left: Arc<dyn ExecutionPlan>,
+ physical_right: Arc<dyn ExecutionPlan>,
+ join_on: join_utils::JoinOn,
+ join_filter: Option<join_utils::JoinFilter>,
+ join_type: &JoinType,
+ null_equality: &datafusion_common::NullEquality,
+) -> Result<Arc<dyn ExecutionPlan>> {
+ // Short-circuit: handle pure cross join (existing behavior)
+ if join_on.is_empty() {
+ if join_filter.is_none() && matches!(join_type, JoinType::Inner) {
+ return Ok(Arc::new(CrossJoinExec::new(physical_left, physical_right)));
+ }
+ }
+
+ // Step 1: Find possible join types for the given Logical Plan
+ // ----------------------------------------------------------------------
+
+ // Build the list of possible algorithms for this join
+ #[derive(Debug, Clone, Copy, PartialEq, Eq)]
+ enum Algo {
+ NLJ,
+ HJ,
+ SMJ,
+ }
+
+ let cfg = &session_state.config_options().optimizer;
+ let can_smj = session_state.config().target_partitions() > 1
+ && session_state.config().repartition_joins();
+
+ let mut possible: Vec<Algo> = Vec::new();
+ if join_on.is_empty() {
+ possible.push(Algo::NLJ);
+ } else {
+ possible.push(Algo::HJ);
+ if can_smj {
+ possible.push(Algo::SMJ);
+ }
+ }
+
+ // Step 2: Filter the possible list according to enable flags from config
+ // ----------------------------------------------------------------------
+
+ // Filter by enable flags
+ let enabled_and_possible: Vec<Algo> = possible
+ .iter()
+ .copied()
+ .filter(|a| match a {
+ Algo::NLJ => cfg.enable_nested_loop_join,
+ Algo::HJ => cfg.enable_hash_join,
+ Algo::SMJ => cfg.enable_sort_merge_join,
+ })
+ .collect();
+
+ if enabled_and_possible.is_empty() {
+ return plan_err!(
+ "No enabled join algorithm is applicable for this join. Possible join types are {:?}. Try to enable them through configurations like `datafusion.optimizer.enable_hash_join`", &possible
+ );
+ }
+
+ // Step 2: Choose and plan the physical join type according to preference
Review Comment:
The comment says 'Step 2' but this is actually Step 3 according to the function documentation above. The comment should be corrected to 'Step 3'.
```suggestion
// Step 3: Choose and plan the physical join type according to preference
```
##########
datafusion/common/src/config.rs:
##########
@@ -822,9 +822,34 @@ config_namespace! {
/// process to reorder the join keys
pub top_down_join_key_reordering: bool, default = true
- /// When set to true, the physical plan optimizer will prefer HashJoin over SortMergeJoin.
+ /// When set to true, the physical plan optimizer will prefer HashJoin when applicable.
+ /// If there are multiple preferred and applicable join types, the optimizer
+ /// will choose one based on heuristics.
/// HashJoin can work more efficiently than SortMergeJoin but consumes more memory
- pub prefer_hash_join: bool, default = true
+ pub prefer_hash_join: bool, default = false
Review Comment:
Changing the default value from `true` to `false` is a breaking change that should be clearly documented in the migration guide. Consider if this default change aligns with the project's backward compatibility policy.
```suggestion
pub prefer_hash_join: bool, default = true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Make join selection configurable through `enable_*` and `prefer_*` options [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on code in PR #17467:
URL: https://github.com/apache/datafusion/pull/17467#discussion_r2329222755
##########
datafusion/common/src/config.rs:
##########
@@ -822,9 +822,34 @@ config_namespace! {
/// process to reorder the join keys
pub top_down_join_key_reordering: bool, default = true
- /// When set to true, the physical plan optimizer will prefer HashJoin over SortMergeJoin.
+ /// When set to true, the physical plan optimizer will prefer HashJoin when applicable.
+ /// If there are multiple preferred and applicable join types, the optimizer
+ /// will choose one based on heuristics.
/// HashJoin can work more efficiently than SortMergeJoin but consumes more memory
- pub prefer_hash_join: bool, default = true
+ pub prefer_hash_join: bool, default = false
Review Comment:
It's not a breaking change. By default there is no join types preferred, and the planner will prioritize HJ according to heuristics, setting it to `true` will just override the planner's default behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Make join selection configurable through `enable_*` and `join_method_priority` options [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #17467:
URL: https://github.com/apache/datafusion/pull/17467#issuecomment-3299789110
FYI I think @NGA-TRAN is planning to propose a more full featured mechanism for generating join plans, which may be related to this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Make join selection configurable through `enable_*` and `join_method_priority` options [datafusion]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17467:
URL: https://github.com/apache/datafusion/pull/17467#discussion_r2346795989
##########
datafusion/core/src/physical_planner/join_planner.rs:
##########
@@ -0,0 +1,212 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use crate::error::Result;
+use crate::execution::context::SessionState;
+use crate::physical_plan::joins::utils as join_utils;
+use crate::physical_plan::joins::{
+ CrossJoinExec, HashJoinExec, NestedLoopJoinExec, PartitionMode, SortMergeJoinExec,
+};
+use crate::physical_plan::ExecutionPlan;
+use arrow::compute::SortOptions;
+use datafusion_common::{config_err, plan_err};
+use datafusion_expr::JoinType;
+
+/// Build the appropriate join `ExecutionPlan` for the given join type, filter, and
+/// configurations.
+///
+/// For example, given an equi-join, the planner may execute it as a Nested Loop
+/// Join, Hash Join, or another strategy. Configuration settings determine which
+/// ExecutionPlan is used.
+///
+/// # Strategy
+/// - Step 1: Find all possible physical join types for the given join logical plan
+/// - No join on keys and no filter => CrossJoin
+/// - With equality? => HJ and SMJ (if with multiple partition)
+/// TODO: The constraint on SMJ is added previously for optimization. Should
+/// we remove it for configurability?
+/// TODO: Allow NLJ for equal join for better configurability.
+/// - Without equality? => NLJ
+/// - Step 2: Filter the possible join types from step 1 according to the configuration
+/// by checking if they're enabled by options like `datafusion.optimizer.enable_hash_join`
+/// - Step 3: Choose one according to the built-in heuristics and also the preference
+/// in the configuration `datafusion.optimizer.join_method_priority`
+pub(super) fn plan_initial_join_exec(
+ session_state: &SessionState,
+ physical_left: Arc<dyn ExecutionPlan>,
+ physical_right: Arc<dyn ExecutionPlan>,
+ join_on: join_utils::JoinOn,
+ join_filter: Option<join_utils::JoinFilter>,
+ join_type: &JoinType,
+ null_equality: &datafusion_common::NullEquality,
+) -> Result<Arc<dyn ExecutionPlan>> {
+ // Short-circuit: handle pure cross join (existing behavior)
+ if join_on.is_empty() && join_filter.is_none() && matches!(join_type, JoinType::Inner)
+ {
+ return Ok(Arc::new(CrossJoinExec::new(physical_left, physical_right)));
+ }
+
+ // Step 1: Find possible join types for the given Logical Plan
+ // ----------------------------------------------------------------------
+
+ // Build the list of possible algorithms for this join
+ #[derive(Debug, Clone, Copy, PartialEq, Eq)]
+ enum Algo {
+ Nlj,
+ Hj,
+ Smj,
+ }
+
+ let cfg = &session_state.config_options().optimizer;
+ let can_smj = session_state.config().target_partitions() > 1
+ && session_state.config().repartition_joins();
+
+ let mut possible: Vec<Algo> = Vec::new();
+ if join_on.is_empty() {
+ possible.push(Algo::Nlj);
+ } else {
+ possible.push(Algo::Hj);
+ if can_smj {
+ possible.push(Algo::Smj);
+ }
+ }
+
+ // Step 2: Filter the possible list according to enable flags from config
+ // ----------------------------------------------------------------------
+
+ // Filter by enable flags
+ let enabled_and_possible: Vec<Algo> = possible
+ .iter()
+ .copied()
+ .filter(|a| match a {
+ Algo::Nlj => cfg.enable_nested_loop_join,
+ Algo::Hj => cfg.enable_hash_join,
+ Algo::Smj => cfg.enable_sort_merge_join,
+ })
+ .collect();
+
+ if enabled_and_possible.is_empty() {
+ return plan_err!(
+ "No enabled join algorithm is applicable for this join. Possible join types are {:?}. Try to enable them through configurations like `datafusion.optimizer.enable_hash_join`", &possible
+ );
+ }
+
+ // Step 3: Choose and plan the physical join type according to
+ // `join_method_priority` and built-in heuristics
+ // ----------------------------------------------------------------------
+
+ // Parse join method priority string into an ordered list of algorithms
+ let parse_priority = |s: &str| -> Result<Vec<Algo>> {
+ let mut out = Vec::new();
+ let mut unknown = Vec::new();
+ for token in s.split(',').map(|t| t.trim()).filter(|t| !t.is_empty()) {
+ match token {
+ "hj" | "hash_join" => out.push(Algo::Hj),
+ "smj" | "sort_merge_join" => out.push(Algo::Smj),
+ "nlj" | "nested_loop_join" => out.push(Algo::Nlj),
+ other => unknown.push(other.to_string()),
+ }
+ }
+ if !unknown.is_empty() {
+ let valid = "hj/hash_join, smj/sort_merge_join, nlj/nested_loop_join";
+ return config_err!(
+ "Invalid join method(s) in datafusion.optimizer.join_method_priority: {}. Valid values: {}",
+ unknown.join(", "),
+ valid
+ );
+ }
+ Ok(out)
+ };
+
+ // Backward compatibility:
+ // If `join_method_priority` is empty, honor legacy `prefer_hash_join` by
+ // setting the priority to a single entry accordingly. Otherwise, parse the
Review Comment:
The parsing logic is executed on every join operation. Since configuration values are typically stable during query execution, consider parsing and caching the priority list once when the configuration is set, rather than re-parsing the string for each join operation.
```suggestion
// Use the cached, pre-parsed join method priority from the configuration.
// Backward compatibility:
// If `join_method_priority` is empty, honor legacy `prefer_hash_join` by
// setting the priority to a single entry accordingly. Otherwise, use the cached, pre-parsed priority list from the configuration.
```
##########
datafusion/common/src/config.rs:
##########
@@ -824,8 +824,35 @@ config_namespace! {
/// When set to true, the physical plan optimizer will prefer HashJoin over SortMergeJoin.
/// HashJoin can work more efficiently than SortMergeJoin but consumes more memory
+ ///
+ /// Note: if `join_method_priority` is set, this configuration will be overridden
+ #[deprecated(since = "51.0.0", note = "Please use configuration option `join_method_priority` instead")]
pub prefer_hash_join: bool, default = true
+ /// Comma-separated join priority (case-insensitive) selecting the first applicable and enabled (through configurations like `enable_hash_join`) join method.
+ ///
+ /// Valid join methods are: hj/hash_join, smj/sort_merge_join, nlj/nested_loop_join.
+ ///
+ /// It is allowed to specify any number of join methods in the configuration. If none of them is applicable, the default join planning heuristic will be used.
+ /// The default join planning policy is: for equi-joins Hash Join will be selected; for non equi-joins, Nested Loop Join will be chosen.
+ ///
+ /// Example usage: SET datafusion.optimizer.join_method_priority = 'hj, nlj'
+ ///
+ /// Note: If this option is not set (default empty string), the deprecated legacy option `prefer_hash_join` will be used.
+ pub join_method_priority: String, transform = str::to_lowercase, default = "".to_string()
Review Comment:
The `str::to_lowercase` transformation is applied on every access of this configuration value. Consider transforming and validating the input once during configuration parsing rather than on every use to avoid repeated string allocations.
```suggestion
pub join_method_priority: String, default = "".to_string()
```
##########
datafusion/core/src/physical_planner/join_planner.rs:
##########
@@ -0,0 +1,212 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use crate::error::Result;
+use crate::execution::context::SessionState;
+use crate::physical_plan::joins::utils as join_utils;
+use crate::physical_plan::joins::{
+ CrossJoinExec, HashJoinExec, NestedLoopJoinExec, PartitionMode, SortMergeJoinExec,
+};
+use crate::physical_plan::ExecutionPlan;
+use arrow::compute::SortOptions;
+use datafusion_common::{config_err, plan_err};
+use datafusion_expr::JoinType;
+
+/// Build the appropriate join `ExecutionPlan` for the given join type, filter, and
+/// configurations.
+///
+/// For example, given an equi-join, the planner may execute it as a Nested Loop
+/// Join, Hash Join, or another strategy. Configuration settings determine which
+/// ExecutionPlan is used.
+///
+/// # Strategy
+/// - Step 1: Find all possible physical join types for the given join logical plan
+/// - No join on keys and no filter => CrossJoin
+/// - With equality? => HJ and SMJ (if with multiple partition)
+/// TODO: The constraint on SMJ is added previously for optimization. Should
+/// we remove it for configurability?
+/// TODO: Allow NLJ for equal join for better configurability.
+/// - Without equality? => NLJ
+/// - Step 2: Filter the possible join types from step 1 according to the configuration
+/// by checking if they're enabled by options like `datafusion.optimizer.enable_hash_join`
+/// - Step 3: Choose one according to the built-in heuristics and also the preference
+/// in the configuration `datafusion.optimizer.join_method_priority`
+pub(super) fn plan_initial_join_exec(
+ session_state: &SessionState,
+ physical_left: Arc<dyn ExecutionPlan>,
+ physical_right: Arc<dyn ExecutionPlan>,
+ join_on: join_utils::JoinOn,
+ join_filter: Option<join_utils::JoinFilter>,
+ join_type: &JoinType,
+ null_equality: &datafusion_common::NullEquality,
+) -> Result<Arc<dyn ExecutionPlan>> {
+ // Short-circuit: handle pure cross join (existing behavior)
+ if join_on.is_empty() && join_filter.is_none() && matches!(join_type, JoinType::Inner)
+ {
+ return Ok(Arc::new(CrossJoinExec::new(physical_left, physical_right)));
+ }
+
+ // Step 1: Find possible join types for the given Logical Plan
+ // ----------------------------------------------------------------------
+
+ // Build the list of possible algorithms for this join
+ #[derive(Debug, Clone, Copy, PartialEq, Eq)]
+ enum Algo {
+ Nlj,
+ Hj,
+ Smj,
+ }
+
+ let cfg = &session_state.config_options().optimizer;
+ let can_smj = session_state.config().target_partitions() > 1
+ && session_state.config().repartition_joins();
+
+ let mut possible: Vec<Algo> = Vec::new();
+ if join_on.is_empty() {
+ possible.push(Algo::Nlj);
+ } else {
+ possible.push(Algo::Hj);
+ if can_smj {
+ possible.push(Algo::Smj);
+ }
+ }
+
+ // Step 2: Filter the possible list according to enable flags from config
+ // ----------------------------------------------------------------------
+
+ // Filter by enable flags
+ let enabled_and_possible: Vec<Algo> = possible
+ .iter()
+ .copied()
+ .filter(|a| match a {
+ Algo::Nlj => cfg.enable_nested_loop_join,
+ Algo::Hj => cfg.enable_hash_join,
+ Algo::Smj => cfg.enable_sort_merge_join,
+ })
+ .collect();
+
+ if enabled_and_possible.is_empty() {
+ return plan_err!(
+ "No enabled join algorithm is applicable for this join. Possible join types are {:?}. Try to enable them through configurations like `datafusion.optimizer.enable_hash_join`", &possible
+ );
+ }
+
+ // Step 3: Choose and plan the physical join type according to
+ // `join_method_priority` and built-in heuristics
+ // ----------------------------------------------------------------------
+
+ // Parse join method priority string into an ordered list of algorithms
+ let parse_priority = |s: &str| -> Result<Vec<Algo>> {
+ let mut out = Vec::new();
+ let mut unknown = Vec::new();
+ for token in s.split(',').map(|t| t.trim()).filter(|t| !t.is_empty()) {
+ match token {
+ "hj" | "hash_join" => out.push(Algo::Hj),
+ "smj" | "sort_merge_join" => out.push(Algo::Smj),
+ "nlj" | "nested_loop_join" => out.push(Algo::Nlj),
Review Comment:
[nitpick] The string matching for join method aliases could benefit from a centralized mapping. Consider defining a constant or function that maps all valid aliases to avoid duplication and make it easier to add new aliases in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Make join selection configurable through `enable_*` and `join_method_priority` options [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17467:
URL: https://github.com/apache/datafusion/pull/17467#issuecomment-3578538225
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org