You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "rkrishn7 (via GitHub)" <gi...@apache.org> on 2025/09/11 22:12:12 UTC
[PR] feat: pushdown hashes hashjoinexec [datafusion]
rkrishn7 opened a new pull request, #17529:
URL: https://github.com/apache/datafusion/pull/17529
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Closes #.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2472639633
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
> I'm not sure if this aligns with your measurements
I think, in terms of latency, the results were similar. The key improvement I noticed was in memory usage -- for instance, Q18 has all distinct left side values, making it the heaviest in terms of memory. There’s a difference from 5.1 GB down to 4.4 GB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: pushdown hashes hashjoinexec [datafusion]
Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2353807675
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
Review Comment:
Thanks @alamb for the diagrams 😄 (from https://github.com/apache/datafusion/issues/7955)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2451923599
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
If it is mainly from having `Vec<Arc<dyn JoinHashMapType>>` (i.e. vec of tables) and doing multiple lookups, maybe we can consider `Arc<dyn JoinHashMapType>` instead?
1. For pushing down 1 table, just keep the original table
2. For >1 tables, merge the tables to one `JoinHashMapType` (with offset to 0).
This will avoid the creation overhead for 1 table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2460947387
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
> I think the way you'd do it is something like (col in hash_table_1) OR (col in hash_table_2) OR (...)
I'm wondering If we do that, wouldn't it result in less aggressive filtering? I don't think we'll get incorrect final result, but more rows would pass the filter than necessary. For example, if probe partition 0 checks ALL build hash tables instead of just build partition 0, it might find hashes from other partitions and let those rows through. But when probe partition 0 later joins with puild Partition 0, those rows won't actually match (since they belong to different partitions). So the final join results will still be correct, but we'll process more rows unnecessarily, reducing the optimization benefit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2466214932
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
Thanks @LiaCastaneda!
Yeah, so this is pretty much exactly what I had done previously. Instead of scanning through the `Vec<Arc<dyn JoinHashMapType>>`, we can leverage the fact that we know the hashing method downstream in `RepartitionExec`. And thus use the same seed/columns used to compute the hash for distributing across partitions.
But, even though we get O(1) lookup in this approach, I found it to be not as performant as the single `HashSet` approach. My thinking is this occurs primarily because we're using any one of the `N` HashMaps on the probe side. So, this likely exhibits much worse cache efficiency than a single HashSet. I'm not sure if this aligns with your measurements
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#issuecomment-3432181634
Benchmarks look promising, but I think we should be able to reduce/remove regressions by avoiding the overhead of creating a new `HashSet` when not needed + some other tweaks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2413960610
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
That makes sense, our plan for setting bounds is to use some available statistics we have ( like row count), and set the `enable_dynamic...` option based on that, so its even before logical and physical DataFusion planning. Would it be possible to do something similar in DataFusion itself while running the optimizer rule?
Although the option you mention is more straightforward, are you suggesting that we set a configuration option for maximum memory consumption and, when building the hashes, stop producing them and free memory if we exceed that limit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#issuecomment-3304163554
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and feat_pushdown-hashes-hashjoinexec
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_pushdown-hashes-hashjoinexec ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │ 2717.61 ms │ 2725.63 ms │ no change │
│ QQuery 1 │ 1351.49 ms │ 1383.95 ms │ no change │
│ QQuery 2 │ 2518.22 ms │ 2567.39 ms │ no change │
│ QQuery 3 │ 1174.45 ms │ 1172.97 ms │ no change │
│ QQuery 4 │ 2266.99 ms │ 2238.41 ms │ no change │
│ QQuery 5 │ 27509.19 ms │ 27546.37 ms │ no change │
│ QQuery 6 │ 4211.49 ms │ 4245.39 ms │ no change │
│ QQuery 7 │ 3546.52 ms │ 3562.03 ms │ no change │
└──────────────┴─────────────┴───────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 45295.95ms │
│ Total Time (feat_pushdown-hashes-hashjoinexec) │ 45442.14ms │
│ Average Time (HEAD) │ 5661.99ms │
│ Average Time (feat_pushdown-hashes-hashjoinexec) │ 5680.27ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 8 │
│ Queries with Failure │ 0 │
└──────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_pushdown-hashes-hashjoinexec ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.15 ms │ 2.22 ms │ no change │
│ QQuery 1 │ 50.36 ms │ 49.92 ms │ no change │
│ QQuery 2 │ 137.12 ms │ 140.45 ms │ no change │
│ QQuery 3 │ 163.55 ms │ 163.97 ms │ no change │
│ QQuery 4 │ 1042.87 ms │ 1079.35 ms │ no change │
│ QQuery 5 │ 1503.48 ms │ 1629.49 ms │ 1.08x slower │
│ QQuery 6 │ 2.20 ms │ 2.17 ms │ no change │
│ QQuery 7 │ 55.16 ms │ 55.88 ms │ no change │
│ QQuery 8 │ 1450.70 ms │ 1483.81 ms │ no change │
│ QQuery 9 │ 1758.13 ms │ 1899.19 ms │ 1.08x slower │
│ QQuery 10 │ 375.94 ms │ 380.50 ms │ no change │
│ QQuery 11 │ 436.65 ms │ 439.74 ms │ no change │
│ QQuery 12 │ 1370.17 ms │ 1395.11 ms │ no change │
│ QQuery 13 │ 2128.89 ms │ 2168.22 ms │ no change │
│ QQuery 14 │ 1264.19 ms │ 1302.04 ms │ no change │
│ QQuery 15 │ 1185.43 ms │ 1262.46 ms │ 1.06x slower │
│ QQuery 16 │ 2628.93 ms │ 2698.18 ms │ no change │
│ QQuery 17 │ 2646.98 ms │ 2684.21 ms │ no change │
│ QQuery 18 │ 5350.80 ms │ 5020.88 ms │ +1.07x faster │
│ QQuery 19 │ 128.21 ms │ 127.70 ms │ no change │
│ QQuery 20 │ 2051.17 ms │ 2077.75 ms │ no change │
│ QQuery 21 │ 2382.70 ms │ 2369.91 ms │ no change │
│ QQuery 22 │ 4069.62 ms │ 4029.18 ms │ no change │
│ QQuery 23 │ 15388.80 ms │ 12911.70 ms │ +1.19x faster │
│ QQuery 24 │ 224.76 ms │ 228.39 ms │ no change │
│ QQuery 25 │ 510.24 ms │ 509.16 ms │ no change │
│ QQuery 26 │ 216.79 ms │ 225.03 ms │ no change │
│ QQuery 27 │ 2983.96 ms │ 2906.25 ms │ no change │
│ QQuery 28 │ 23267.75 ms │ 23351.25 ms │ no change │
│ QQuery 29 │ 963.44 ms │ 1019.77 ms │ 1.06x slower │
│ QQuery 30 │ 1349.04 ms │ 1326.20 ms │ no change │
│ QQuery 31 │ 1350.83 ms │ 1335.29 ms │ no change │
│ QQuery 32 │ 4846.62 ms │ 4877.66 ms │ no change │
│ QQuery 33 │ 5932.21 ms │ 5797.16 ms │ no change │
│ QQuery 34 │ 6005.87 ms │ 6076.10 ms │ no change │
│ QQuery 35 │ 2078.60 ms │ 2080.23 ms │ no change │
│ QQuery 36 │ 124.39 ms │ 122.41 ms │ no change │
│ QQuery 37 │ 55.88 ms │ 54.30 ms │ no change │
│ QQuery 38 │ 125.99 ms │ 126.27 ms │ no change │
│ QQuery 39 │ 208.15 ms │ 202.62 ms │ no change │
│ QQuery 40 │ 44.99 ms │ 43.80 ms │ no change │
│ QQuery 41 │ 42.19 ms │ 42.26 ms │ no change │
│ QQuery 42 │ 33.63 ms │ 33.99 ms │ no change │
└──────────────┴─────────────┴───────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 97939.49ms │
│ Total Time (feat_pushdown-hashes-hashjoinexec) │ 95732.17ms │
│ Average Time (HEAD) │ 2277.66ms │
│ Average Time (feat_pushdown-hashes-hashjoinexec) │ 2226.33ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 4 │
│ Queries with No Change │ 37 │
│ Queries with Failure │ 0 │
└──────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ feat_pushdown-hashes-hashjoinexec ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 170.84 ms │ 168.69 ms │ no change │
│ QQuery 2 │ 27.69 ms │ 27.60 ms │ no change │
│ QQuery 3 │ 45.32 ms │ 46.66 ms │ no change │
│ QQuery 4 │ 27.50 ms │ 27.10 ms │ no change │
│ QQuery 5 │ 75.04 ms │ 77.54 ms │ no change │
│ QQuery 6 │ 19.38 ms │ 19.93 ms │ no change │
│ QQuery 7 │ 148.69 ms │ 148.04 ms │ no change │
│ QQuery 8 │ 38.63 ms │ 32.15 ms │ +1.20x faster │
│ QQuery 9 │ 103.45 ms │ 86.18 ms │ +1.20x faster │
│ QQuery 10 │ 59.85 ms │ 59.21 ms │ no change │
│ QQuery 11 │ 41.71 ms │ 40.99 ms │ no change │
│ QQuery 12 │ 51.25 ms │ 50.80 ms │ no change │
│ QQuery 13 │ 47.90 ms │ 47.13 ms │ no change │
│ QQuery 14 │ 14.05 ms │ 14.06 ms │ no change │
│ QQuery 15 │ 24.19 ms │ 24.27 ms │ no change │
│ QQuery 16 │ 24.75 ms │ 23.96 ms │ no change │
│ QQuery 17 │ 146.04 ms │ 147.61 ms │ no change │
│ QQuery 18 │ 331.51 ms │ 330.26 ms │ no change │
│ QQuery 19 │ 36.24 ms │ 36.63 ms │ no change │
│ QQuery 20 │ 48.12 ms │ 48.99 ms │ no change │
│ QQuery 21 │ 218.30 ms │ 228.72 ms │ no change │
│ QQuery 22 │ 19.70 ms │ 19.98 ms │ no change │
└──────────────┴───────────┴───────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1720.15ms │
│ Total Time (feat_pushdown-hashes-hashjoinexec) │ 1706.50ms │
│ Average Time (HEAD) │ 78.19ms │
│ Average Time (feat_pushdown-hashes-hashjoinexec) │ 77.57ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 20 │
│ Queries with Failure │ 0 │
└──────────────────────────────────────────────────┴───────────┘
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2356316333
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
I think the way you'd do it is something like `(col in hash_table_1) OR (col in hash_table_2) OR (...)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2451933763
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
+ /// Memory reservation tracking the memory used by `left_hashes`
+ reservation: MemoryReservation,
+}
+
+impl SharedBuildAccumulator {
+ /// Creates a new [SharedBuildAccumulator] configured for the given partition mode
+ ///
+ /// This method calculates how many times `collect_build_side` will be called based on the
+ /// partition mode's execution pattern. This count is critical for determining when we have
+ /// complete information from all partitions to build the dynamic filter.
+ ///
+ /// ## Partition Mode Execution Patterns
+ ///
+ /// - **CollectLeft**: Build side is collected ONCE from partition 0 and shared via `OnceFut`
+ /// across all output partitions. Each output partition calls `collect_build_side` to access the shared build data.
+ /// Although this results in multiple invocations, the `report_partition_bounds` function contains deduplication logic to handle them safely.
+ /// Expected calls = number of output partitions.
+ ///
+ ///
+ /// - **Partitioned**: Each partition independently builds its own hash table by calling
+ /// `collect_build_side` once. Expected calls = number of build partitions.
+ ///
+ /// - **Auto**: Placeholder mode resolved during optimization. Uses 1 as safe default since
+ /// the actual mode will be determined and a new bounds_accumulator created before execution.
+ ///
+ /// ## Why This Matters
+ ///
+ /// We cannot build a partial filter from some partitions - it would incorrectly eliminate
+ /// valid join results. We must wait until we have complete bounds information from ALL
+ /// relevant partitions before updating the dynamic filter.
+ pub(crate) fn new_from_partition_mode(
+ partition_mode: PartitionMode,
+ left_child: &dyn ExecutionPlan,
+ right_child: &dyn ExecutionPlan,
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ random_state: &'static RandomState,
+ reservation: MemoryReservation,
+ ) -> Self {
+ // Troubleshooting: If partition counts are incorrect, verify this logic matches
+ // the actual execution pattern in collect_build_side()
+ let expected_calls = match partition_mode {
+ // Each output partition accesses shared build data
+ PartitionMode::CollectLeft => {
+ right_child.output_partitioning().partition_count()
+ }
+ // Each partition builds its own data
+ PartitionMode::Partitioned => {
+ left_child.output_partitioning().partition_count()
+ }
+ // Default value, will be resolved during optimization (does not exist once `execute()` is called; will be replaced by one of the other two)
+ PartitionMode::Auto => unreachable!("PartitionMode::Auto should not be present at execution time. This is a bug in DataFusion, please report it!"),
+ };
+ Self {
+ inner: Mutex::new(SharedBuildState {
+ bounds: Vec::with_capacity(expected_calls),
+ left_hashes: HashSet::with_hasher(core::hash::BuildHasherDefault::<
+ NoHashHasher,
+ >::default()),
+ reservation,
+ }),
+ barrier: Barrier::new(expected_calls),
+ dynamic_filter,
+ on_right,
+ random_state,
+ }
+ }
+
+ /// Create a filter expression from individual partition bounds using OR logic.
+ ///
+ /// This creates a filter where each partition's bounds form a conjunction (AND)
+ /// of column range predicates, and all partitions are combined with OR.
+ ///
+ /// For example, with 2 partitions and 2 columns:
+ /// ((col0 >= p0_min0 AND col0 <= p0_max0 AND col1 >= p0_min1 AND col1 <= p0_max1)
+ /// OR
+ /// (col0 >= p1_min0 AND col0 <= p1_max0 AND col1 >= p1_min1 AND col1 <= p1_max1))
+ pub(crate) fn create_filter_from_partition_bounds(
+ &self,
+ bounds: &[PartitionBounds],
+ ) -> Result<Arc<dyn PhysicalExpr>> {
+ if bounds.is_empty() {
+ return Ok(lit(true));
+ }
+
+ // Create a predicate for each partition
+ let mut partition_predicates = Vec::with_capacity(bounds.len());
+
+ for partition_bounds in bounds.iter().sorted_by_key(|b| b.partition) {
+ // Create range predicates for each join key in this partition
+ let mut column_predicates = Vec::with_capacity(partition_bounds.len());
+
+ for (col_idx, right_expr) in self.on_right.iter().enumerate() {
+ if let Some(column_bounds) = partition_bounds.get_column_bounds(col_idx) {
+ // Create predicate: col >= min AND col <= max
+ let min_expr = Arc::new(BinaryExpr::new(
+ Arc::clone(right_expr),
+ Operator::GtEq,
+ lit(column_bounds.min.clone()),
+ )) as Arc<dyn PhysicalExpr>;
+ let max_expr = Arc::new(BinaryExpr::new(
+ Arc::clone(right_expr),
+ Operator::LtEq,
+ lit(column_bounds.max.clone()),
+ )) as Arc<dyn PhysicalExpr>;
+ let range_expr =
+ Arc::new(BinaryExpr::new(min_expr, Operator::And, max_expr))
+ as Arc<dyn PhysicalExpr>;
+ column_predicates.push(range_expr);
+ }
+ }
+
+ // Combine all column predicates for this partition with AND
+ if !column_predicates.is_empty() {
+ let partition_predicate = column_predicates
+ .into_iter()
+ .reduce(|acc, pred| {
+ Arc::new(BinaryExpr::new(acc, Operator::And, pred))
+ as Arc<dyn PhysicalExpr>
+ })
+ .unwrap();
+ partition_predicates.push(partition_predicate);
+ }
+ }
+
+ // Combine all partition predicates with OR
+ let combined_predicate = partition_predicates
+ .into_iter()
+ .reduce(|acc, pred| {
+ Arc::new(BinaryExpr::new(acc, Operator::Or, pred))
+ as Arc<dyn PhysicalExpr>
+ })
+ .unwrap_or_else(|| lit(true));
+
+ Ok(combined_predicate)
+ }
+
+ /// Report information from a completed partition and update dynamic filter if all partitions are done
+ ///
+ /// This method coordinates the dynamic filter updates across all partitions. It stores the
+ /// information from the current partition, increments the completion counter, and when all
+ /// partitions have reported, updates the dynamic filter with the appropriate expressions.
+ ///
+ /// This method is async and uses a [`tokio::sync::Barrier`] to wait for all partitions
+ /// to report their bounds. Once that occurs, the method will resolve for all callers and the
+ /// dynamic filter will be updated exactly once.
+ ///
+ /// # Note
+ ///
+ /// As barriers are reusable, it is likely an error to call this method more times than the
+ /// total number of partitions - as it can lead to pending futures that never resolve. We rely
+ /// on correct usage from the caller rather than imposing additional checks here. If this is a concern,
+ /// consider making the resulting future shared so the ready result can be reused.
+ ///
+ /// # Arguments
+ /// * `left_side_partition_id` - The identifier for the **left-side** partition reporting its bounds
+ /// * `partition_bounds` - The bounds computed by this partition (if enabled)
+ /// * `left_hash_map` - The hashes from the build side for this partition (if enabled)
+ ///
+ /// # Returns
+ /// * `Result<()>` - Ok if successful, Err if filter update failed
+ pub(crate) async fn report_information(
+ &self,
+ left_side_partition_id: usize,
+ partition_bounds: Option<Vec<ColumnBounds>>,
+ left_hash_map: Option<Arc<dyn JoinHashMapType>>,
+ ) -> Result<()> {
+ // Store bounds in the accumulator - this runs once per partition
+ if let Some(bounds) = partition_bounds {
+ let mut guard = self.inner.lock();
+
+ let should_push = if let Some(last_bound) = guard.bounds.last() {
+ // In `PartitionMode::CollectLeft`, all streams on the left side share the same partition id (0).
+ // Since this function can be called multiple times for that same partition, we must deduplicate
+ // by checking against the last recorded bound.
+ last_bound.partition != left_side_partition_id
+ } else {
+ true
+ };
+
+ if should_push {
+ guard
+ .bounds
+ .push(PartitionBounds::new(left_side_partition_id, bounds));
+ }
+ }
+
+ if let Some(left_hash_map) = left_hash_map {
+ if left_hash_map.num_hashes() > 0 {
+ let mut inner = self.inner.lock();
+
+ let fixed_size = size_of::<NoHashSet<u64>>();
+
+ let estimated_additional_size =
+ estimate_memory_size::<u64>(left_hash_map.num_hashes(), fixed_size)?;
+ inner.reservation.try_grow(estimated_additional_size)?;
+ inner.left_hashes.extend(left_hash_map.hashes());
Review Comment:
Can we maybe derive the capacity for `left_hashes` upfront to reduce the cost of building the map?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2413960610
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
That makes sense, our current plan for using this API is to use some available statistics we have ( like row count), and set the `enable_dynamic...` option based on that, so its even before logical and physical DataFusion planning. Would it be possible to do something similar in DataFusion itself while running the optimizer rule?
Although the option you mention is more straightforward, are you suggesting that we set a configuration option for maximum memory consumption and, when building the hashes, stop producing them and free memory if we exceed that limit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#issuecomment-3300577436
@adriangb I think this is probably ready for an initial look when you get a chance! I plan on adding unit + fuzz tests as well. But let me know if you have any other thoughts re: testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2466292485
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
But I do think @adriangb's thinking is correct in his comment [here](https://github.com/apache/datafusion/pull/17529#issuecomment-3379650299). Even if this approach is slightly less performant than allocating an entire new `HashSet`, it probably wins just on account of no extra memory overhead.
Interestingly I had thought the `HashSet` approach yielded substantially better results but looking at the results in your branch @LiaCastaneda they seem to be somewhat comparable? I will test out again today locally so we have another comparison
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#issuecomment-3838510793
@rkrishn7 I think this work has been superseded and this PR now has many conflicts - should we close it? Sorry we never got to include your work...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2451899914
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
+ /// Memory reservation tracking the memory used by `left_hashes`
+ reservation: MemoryReservation,
+}
+
+impl SharedBuildAccumulator {
+ /// Creates a new [SharedBuildAccumulator] configured for the given partition mode
+ ///
+ /// This method calculates how many times `collect_build_side` will be called based on the
+ /// partition mode's execution pattern. This count is critical for determining when we have
+ /// complete information from all partitions to build the dynamic filter.
+ ///
+ /// ## Partition Mode Execution Patterns
+ ///
+ /// - **CollectLeft**: Build side is collected ONCE from partition 0 and shared via `OnceFut`
+ /// across all output partitions. Each output partition calls `collect_build_side` to access the shared build data.
+ /// Although this results in multiple invocations, the `report_partition_bounds` function contains deduplication logic to handle them safely.
+ /// Expected calls = number of output partitions.
+ ///
+ ///
+ /// - **Partitioned**: Each partition independently builds its own hash table by calling
+ /// `collect_build_side` once. Expected calls = number of build partitions.
+ ///
+ /// - **Auto**: Placeholder mode resolved during optimization. Uses 1 as safe default since
+ /// the actual mode will be determined and a new bounds_accumulator created before execution.
+ ///
+ /// ## Why This Matters
+ ///
+ /// We cannot build a partial filter from some partitions - it would incorrectly eliminate
+ /// valid join results. We must wait until we have complete bounds information from ALL
+ /// relevant partitions before updating the dynamic filter.
+ pub(crate) fn new_from_partition_mode(
+ partition_mode: PartitionMode,
+ left_child: &dyn ExecutionPlan,
+ right_child: &dyn ExecutionPlan,
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ random_state: &'static RandomState,
+ reservation: MemoryReservation,
+ ) -> Self {
+ // Troubleshooting: If partition counts are incorrect, verify this logic matches
+ // the actual execution pattern in collect_build_side()
+ let expected_calls = match partition_mode {
+ // Each output partition accesses shared build data
+ PartitionMode::CollectLeft => {
+ right_child.output_partitioning().partition_count()
+ }
+ // Each partition builds its own data
+ PartitionMode::Partitioned => {
+ left_child.output_partitioning().partition_count()
+ }
+ // Default value, will be resolved during optimization (does not exist once `execute()` is called; will be replaced by one of the other two)
+ PartitionMode::Auto => unreachable!("PartitionMode::Auto should not be present at execution time. This is a bug in DataFusion, please report it!"),
+ };
+ Self {
+ inner: Mutex::new(SharedBuildState {
+ bounds: Vec::with_capacity(expected_calls),
+ left_hashes: HashSet::with_hasher(core::hash::BuildHasherDefault::<
+ NoHashHasher,
+ >::default()),
+ reservation,
+ }),
+ barrier: Barrier::new(expected_calls),
+ dynamic_filter,
+ on_right,
+ random_state,
+ }
+ }
+
+ /// Create a filter expression from individual partition bounds using OR logic.
+ ///
+ /// This creates a filter where each partition's bounds form a conjunction (AND)
+ /// of column range predicates, and all partitions are combined with OR.
+ ///
+ /// For example, with 2 partitions and 2 columns:
+ /// ((col0 >= p0_min0 AND col0 <= p0_max0 AND col1 >= p0_min1 AND col1 <= p0_max1)
+ /// OR
+ /// (col0 >= p1_min0 AND col0 <= p1_max0 AND col1 >= p1_min1 AND col1 <= p1_max1))
+ pub(crate) fn create_filter_from_partition_bounds(
+ &self,
+ bounds: &[PartitionBounds],
+ ) -> Result<Arc<dyn PhysicalExpr>> {
+ if bounds.is_empty() {
+ return Ok(lit(true));
+ }
+
+ // Create a predicate for each partition
+ let mut partition_predicates = Vec::with_capacity(bounds.len());
+
+ for partition_bounds in bounds.iter().sorted_by_key(|b| b.partition) {
+ // Create range predicates for each join key in this partition
+ let mut column_predicates = Vec::with_capacity(partition_bounds.len());
+
+ for (col_idx, right_expr) in self.on_right.iter().enumerate() {
+ if let Some(column_bounds) = partition_bounds.get_column_bounds(col_idx) {
+ // Create predicate: col >= min AND col <= max
+ let min_expr = Arc::new(BinaryExpr::new(
+ Arc::clone(right_expr),
+ Operator::GtEq,
+ lit(column_bounds.min.clone()),
+ )) as Arc<dyn PhysicalExpr>;
+ let max_expr = Arc::new(BinaryExpr::new(
+ Arc::clone(right_expr),
+ Operator::LtEq,
+ lit(column_bounds.max.clone()),
+ )) as Arc<dyn PhysicalExpr>;
+ let range_expr =
+ Arc::new(BinaryExpr::new(min_expr, Operator::And, max_expr))
+ as Arc<dyn PhysicalExpr>;
+ column_predicates.push(range_expr);
+ }
+ }
+
+ // Combine all column predicates for this partition with AND
+ if !column_predicates.is_empty() {
+ let partition_predicate = column_predicates
+ .into_iter()
+ .reduce(|acc, pred| {
+ Arc::new(BinaryExpr::new(acc, Operator::And, pred))
+ as Arc<dyn PhysicalExpr>
+ })
+ .unwrap();
+ partition_predicates.push(partition_predicate);
+ }
+ }
+
+ // Combine all partition predicates with OR
+ let combined_predicate = partition_predicates
+ .into_iter()
+ .reduce(|acc, pred| {
+ Arc::new(BinaryExpr::new(acc, Operator::Or, pred))
+ as Arc<dyn PhysicalExpr>
+ })
+ .unwrap_or_else(|| lit(true));
+
+ Ok(combined_predicate)
+ }
+
+ /// Report information from a completed partition and update dynamic filter if all partitions are done
+ ///
+ /// This method coordinates the dynamic filter updates across all partitions. It stores the
+ /// information from the current partition, increments the completion counter, and when all
+ /// partitions have reported, updates the dynamic filter with the appropriate expressions.
+ ///
+ /// This method is async and uses a [`tokio::sync::Barrier`] to wait for all partitions
+ /// to report their bounds. Once that occurs, the method will resolve for all callers and the
+ /// dynamic filter will be updated exactly once.
+ ///
+ /// # Note
+ ///
+ /// As barriers are reusable, it is likely an error to call this method more times than the
+ /// total number of partitions - as it can lead to pending futures that never resolve. We rely
+ /// on correct usage from the caller rather than imposing additional checks here. If this is a concern,
+ /// consider making the resulting future shared so the ready result can be reused.
+ ///
+ /// # Arguments
+ /// * `left_side_partition_id` - The identifier for the **left-side** partition reporting its bounds
+ /// * `partition_bounds` - The bounds computed by this partition (if enabled)
+ /// * `left_hash_map` - The hashes from the build side for this partition (if enabled)
+ ///
+ /// # Returns
+ /// * `Result<()>` - Ok if successful, Err if filter update failed
+ pub(crate) async fn report_information(
+ &self,
+ left_side_partition_id: usize,
+ partition_bounds: Option<Vec<ColumnBounds>>,
+ left_hash_map: Option<Arc<dyn JoinHashMapType>>,
+ ) -> Result<()> {
+ // Store bounds in the accumulator - this runs once per partition
+ if let Some(bounds) = partition_bounds {
+ let mut guard = self.inner.lock();
+
+ let should_push = if let Some(last_bound) = guard.bounds.last() {
+ // In `PartitionMode::CollectLeft`, all streams on the left side share the same partition id (0).
+ // Since this function can be called multiple times for that same partition, we must deduplicate
+ // by checking against the last recorded bound.
+ last_bound.partition != left_side_partition_id
+ } else {
+ true
+ };
+
+ if should_push {
+ guard
+ .bounds
+ .push(PartitionBounds::new(left_side_partition_id, bounds));
+ }
+ }
+
+ if let Some(left_hash_map) = left_hash_map {
+ if left_hash_map.num_hashes() > 0 {
+ let mut inner = self.inner.lock();
+
+ let fixed_size = size_of::<NoHashSet<u64>>();
+
+ let estimated_additional_size =
+ estimate_memory_size::<u64>(left_hash_map.num_hashes(), fixed_size)?;
+ inner.reservation.try_grow(estimated_additional_size)?;
+ inner.left_hashes.extend(left_hash_map.hashes());
+ }
+ }
+
+ if self.barrier.wait().await.is_leader() {
+ // All partitions have reported, so we can update the filter
+ let mut inner = self.inner.lock();
+ let maybe_bounds_expr = if !inner.bounds.is_empty() {
+ Some(self.create_filter_from_partition_bounds(&inner.bounds)?)
+ } else {
+ None
+ };
+
+ let maybe_hash_eval_expr = if !inner.left_hashes.is_empty() {
+ Some(Arc::new(HashComparePhysicalExpr::new(
+ self.on_right.clone(),
+ Arc::new(std::mem::take(&mut inner.left_hashes)),
+ self.random_state,
+ inner.reservation.take(),
+ )) as Arc<dyn PhysicalExpr>)
+ } else {
+ None
+ };
+
+ let filter_expr = match (maybe_bounds_expr, maybe_hash_eval_expr) {
+ (None, Some(expr)) | (Some(expr), None) => expr,
+ // Bounds collection and hash collection are current mutually exclusive, so this branch should not be
+ // taken at the moment. However, if they are ever enabled together, this is how we would combine them.
+ (Some(bounds_expr), Some(hash_eval_expr)) => Arc::new(BinaryExpr::new(
+ bounds_expr,
+ Operator::And,
+ Arc::clone(&hash_eval_expr),
+ )),
+ _ => return Ok(()),
+ };
+
+ self.dynamic_filter.update(filter_expr)?;
+ }
+
+ Ok(())
+ }
+}
+
+impl fmt::Debug for SharedBuildAccumulator {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ write!(f, "SharedBuildAccumulator")
+ }
+}
+
+/// A [`PhysicalExpr`] that evaluates to a boolean array indicating which rows in a batch
+/// have hashes that exist in a given set of hashes.
+///
+/// This is currently used to implement hash-based dynamic filters in hash joins. That is,
+/// this expression can be pushed down to the probe side of a hash join to filter out rows
+/// that do not have a matching hash in the build side, allowing the scan to emit only the rows
+/// that are guaranteed to have at least one match.
+struct HashComparePhysicalExpr {
+ /// Expressions that will be evaluated to compute hashes for filtering
+ exprs: Vec<Arc<dyn PhysicalExpr>>,
+ /// Hashes to filter against
+ hashes: Arc<NoHashSet<u64>>,
+ /// Random state for hash computation
+ random_state: &'static RandomState,
+ /// Memory reservation used to track the memory used by `hashes`
+ reservation: MemoryReservation,
+}
+
+impl HashComparePhysicalExpr {
+ pub fn new(
+ exprs: Vec<Arc<dyn PhysicalExpr>>,
+ hashes: Arc<NoHashSet<u64>>,
+ random_state: &'static RandomState,
+ reservation: MemoryReservation,
+ ) -> Self {
+ Self {
+ exprs,
+ hashes,
+ random_state,
+ reservation,
+ }
+ }
+}
+
+impl fmt::Debug for HashComparePhysicalExpr {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ write!(f, "HashComparePhysicalExpr [ {:?} ]", self.exprs)
+ }
+}
+
+impl Hash for HashComparePhysicalExpr {
+ fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
+ self.exprs.hash(state);
+ }
+}
+
+impl PartialEq for HashComparePhysicalExpr {
+ fn eq(&self, other: &Self) -> bool {
+ // TODO: We limit comparison to uphold the equality property w.r.t `Hash`.
+ // However, this means we may consider two expressions equal when they are actually not.
+ // Since this is currently just used internally for dynamic filters, this may be acceptable
+ // for now. If this expression is ever exposed more broadly, we should revisit this.
+ self.exprs == other.exprs
+ }
+}
+
+impl Eq for HashComparePhysicalExpr {}
+
+impl fmt::Display for HashComparePhysicalExpr {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ let exprs = self
+ .exprs
+ .iter()
+ .map(|e| e.to_string())
+ .collect::<Vec<_>>()
+ .join(", ");
+ write!(f, "HashComparePhysicalExpr [ {exprs} ]")
+ }
+}
+
+impl PhysicalExpr for HashComparePhysicalExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ self.exprs.iter().collect()
+ }
+
+ fn with_new_children(
+ self: Arc<Self>,
+ children: Vec<Arc<dyn PhysicalExpr>>,
+ ) -> Result<Arc<dyn PhysicalExpr>> {
+ Ok(Arc::new(Self {
+ exprs: children,
+ hashes: Arc::clone(&self.hashes),
+ random_state: self.random_state,
+ reservation: self.reservation.new_empty(),
+ }))
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(DataType::Boolean)
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(false)
+ }
+
+ fn evaluate(
+ &self,
+ batch: &arrow::record_batch::RecordBatch,
+ ) -> Result<ColumnarValue> {
+ let num_rows = batch.num_rows();
+
+ let expr_values = self
+ .exprs
+ .iter()
+ .map(|col| col.evaluate(batch)?.into_array(num_rows))
+ .collect::<Result<Vec<_>>>()?;
+
+ // Compute hashes for each row based on the evaluated expressions
+ let mut hashes_buffer = vec![0; num_rows];
+ create_hashes(&expr_values, self.random_state, &mut hashes_buffer)?;
+
+ // Create a boolean array where each position indicates if the corresponding hash is in the set of known hashes
+ let mut buf = MutableBuffer::from_len_zeroed(bit_util::ceil(num_rows, 8));
Review Comment:
Should be able to use `MutableBuffer::collect_bool`
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
+ /// Memory reservation tracking the memory used by `left_hashes`
+ reservation: MemoryReservation,
+}
+
+impl SharedBuildAccumulator {
+ /// Creates a new [SharedBuildAccumulator] configured for the given partition mode
+ ///
+ /// This method calculates how many times `collect_build_side` will be called based on the
+ /// partition mode's execution pattern. This count is critical for determining when we have
+ /// complete information from all partitions to build the dynamic filter.
+ ///
+ /// ## Partition Mode Execution Patterns
+ ///
+ /// - **CollectLeft**: Build side is collected ONCE from partition 0 and shared via `OnceFut`
+ /// across all output partitions. Each output partition calls `collect_build_side` to access the shared build data.
+ /// Although this results in multiple invocations, the `report_partition_bounds` function contains deduplication logic to handle them safely.
+ /// Expected calls = number of output partitions.
+ ///
+ ///
+ /// - **Partitioned**: Each partition independently builds its own hash table by calling
+ /// `collect_build_side` once. Expected calls = number of build partitions.
+ ///
+ /// - **Auto**: Placeholder mode resolved during optimization. Uses 1 as safe default since
+ /// the actual mode will be determined and a new bounds_accumulator created before execution.
+ ///
+ /// ## Why This Matters
+ ///
+ /// We cannot build a partial filter from some partitions - it would incorrectly eliminate
+ /// valid join results. We must wait until we have complete bounds information from ALL
+ /// relevant partitions before updating the dynamic filter.
+ pub(crate) fn new_from_partition_mode(
+ partition_mode: PartitionMode,
+ left_child: &dyn ExecutionPlan,
+ right_child: &dyn ExecutionPlan,
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ random_state: &'static RandomState,
+ reservation: MemoryReservation,
+ ) -> Self {
+ // Troubleshooting: If partition counts are incorrect, verify this logic matches
+ // the actual execution pattern in collect_build_side()
+ let expected_calls = match partition_mode {
+ // Each output partition accesses shared build data
+ PartitionMode::CollectLeft => {
+ right_child.output_partitioning().partition_count()
+ }
+ // Each partition builds its own data
+ PartitionMode::Partitioned => {
+ left_child.output_partitioning().partition_count()
+ }
+ // Default value, will be resolved during optimization (does not exist once `execute()` is called; will be replaced by one of the other two)
+ PartitionMode::Auto => unreachable!("PartitionMode::Auto should not be present at execution time. This is a bug in DataFusion, please report it!"),
+ };
+ Self {
+ inner: Mutex::new(SharedBuildState {
+ bounds: Vec::with_capacity(expected_calls),
+ left_hashes: HashSet::with_hasher(core::hash::BuildHasherDefault::<
+ NoHashHasher,
+ >::default()),
+ reservation,
+ }),
+ barrier: Barrier::new(expected_calls),
+ dynamic_filter,
+ on_right,
+ random_state,
+ }
+ }
+
+ /// Create a filter expression from individual partition bounds using OR logic.
+ ///
+ /// This creates a filter where each partition's bounds form a conjunction (AND)
+ /// of column range predicates, and all partitions are combined with OR.
+ ///
+ /// For example, with 2 partitions and 2 columns:
+ /// ((col0 >= p0_min0 AND col0 <= p0_max0 AND col1 >= p0_min1 AND col1 <= p0_max1)
+ /// OR
+ /// (col0 >= p1_min0 AND col0 <= p1_max0 AND col1 >= p1_min1 AND col1 <= p1_max1))
+ pub(crate) fn create_filter_from_partition_bounds(
+ &self,
+ bounds: &[PartitionBounds],
+ ) -> Result<Arc<dyn PhysicalExpr>> {
+ if bounds.is_empty() {
+ return Ok(lit(true));
+ }
+
+ // Create a predicate for each partition
+ let mut partition_predicates = Vec::with_capacity(bounds.len());
+
+ for partition_bounds in bounds.iter().sorted_by_key(|b| b.partition) {
+ // Create range predicates for each join key in this partition
+ let mut column_predicates = Vec::with_capacity(partition_bounds.len());
+
+ for (col_idx, right_expr) in self.on_right.iter().enumerate() {
+ if let Some(column_bounds) = partition_bounds.get_column_bounds(col_idx) {
+ // Create predicate: col >= min AND col <= max
+ let min_expr = Arc::new(BinaryExpr::new(
+ Arc::clone(right_expr),
+ Operator::GtEq,
+ lit(column_bounds.min.clone()),
+ )) as Arc<dyn PhysicalExpr>;
+ let max_expr = Arc::new(BinaryExpr::new(
+ Arc::clone(right_expr),
+ Operator::LtEq,
+ lit(column_bounds.max.clone()),
+ )) as Arc<dyn PhysicalExpr>;
+ let range_expr =
+ Arc::new(BinaryExpr::new(min_expr, Operator::And, max_expr))
+ as Arc<dyn PhysicalExpr>;
+ column_predicates.push(range_expr);
+ }
+ }
+
+ // Combine all column predicates for this partition with AND
+ if !column_predicates.is_empty() {
+ let partition_predicate = column_predicates
+ .into_iter()
+ .reduce(|acc, pred| {
+ Arc::new(BinaryExpr::new(acc, Operator::And, pred))
+ as Arc<dyn PhysicalExpr>
+ })
+ .unwrap();
+ partition_predicates.push(partition_predicate);
+ }
+ }
+
+ // Combine all partition predicates with OR
+ let combined_predicate = partition_predicates
+ .into_iter()
+ .reduce(|acc, pred| {
+ Arc::new(BinaryExpr::new(acc, Operator::Or, pred))
+ as Arc<dyn PhysicalExpr>
+ })
+ .unwrap_or_else(|| lit(true));
+
+ Ok(combined_predicate)
+ }
+
+ /// Report information from a completed partition and update dynamic filter if all partitions are done
+ ///
+ /// This method coordinates the dynamic filter updates across all partitions. It stores the
+ /// information from the current partition, increments the completion counter, and when all
+ /// partitions have reported, updates the dynamic filter with the appropriate expressions.
+ ///
+ /// This method is async and uses a [`tokio::sync::Barrier`] to wait for all partitions
+ /// to report their bounds. Once that occurs, the method will resolve for all callers and the
+ /// dynamic filter will be updated exactly once.
+ ///
+ /// # Note
+ ///
+ /// As barriers are reusable, it is likely an error to call this method more times than the
+ /// total number of partitions - as it can lead to pending futures that never resolve. We rely
+ /// on correct usage from the caller rather than imposing additional checks here. If this is a concern,
+ /// consider making the resulting future shared so the ready result can be reused.
+ ///
+ /// # Arguments
+ /// * `left_side_partition_id` - The identifier for the **left-side** partition reporting its bounds
+ /// * `partition_bounds` - The bounds computed by this partition (if enabled)
+ /// * `left_hash_map` - The hashes from the build side for this partition (if enabled)
+ ///
+ /// # Returns
+ /// * `Result<()>` - Ok if successful, Err if filter update failed
+ pub(crate) async fn report_information(
+ &self,
+ left_side_partition_id: usize,
+ partition_bounds: Option<Vec<ColumnBounds>>,
+ left_hash_map: Option<Arc<dyn JoinHashMapType>>,
+ ) -> Result<()> {
+ // Store bounds in the accumulator - this runs once per partition
+ if let Some(bounds) = partition_bounds {
+ let mut guard = self.inner.lock();
+
+ let should_push = if let Some(last_bound) = guard.bounds.last() {
+ // In `PartitionMode::CollectLeft`, all streams on the left side share the same partition id (0).
+ // Since this function can be called multiple times for that same partition, we must deduplicate
+ // by checking against the last recorded bound.
+ last_bound.partition != left_side_partition_id
+ } else {
+ true
+ };
+
+ if should_push {
+ guard
+ .bounds
+ .push(PartitionBounds::new(left_side_partition_id, bounds));
+ }
+ }
+
+ if let Some(left_hash_map) = left_hash_map {
+ if left_hash_map.num_hashes() > 0 {
+ let mut inner = self.inner.lock();
+
+ let fixed_size = size_of::<NoHashSet<u64>>();
+
+ let estimated_additional_size =
+ estimate_memory_size::<u64>(left_hash_map.num_hashes(), fixed_size)?;
+ inner.reservation.try_grow(estimated_additional_size)?;
+ inner.left_hashes.extend(left_hash_map.hashes());
+ }
+ }
+
+ if self.barrier.wait().await.is_leader() {
+ // All partitions have reported, so we can update the filter
+ let mut inner = self.inner.lock();
+ let maybe_bounds_expr = if !inner.bounds.is_empty() {
+ Some(self.create_filter_from_partition_bounds(&inner.bounds)?)
+ } else {
+ None
+ };
+
+ let maybe_hash_eval_expr = if !inner.left_hashes.is_empty() {
+ Some(Arc::new(HashComparePhysicalExpr::new(
+ self.on_right.clone(),
+ Arc::new(std::mem::take(&mut inner.left_hashes)),
+ self.random_state,
+ inner.reservation.take(),
+ )) as Arc<dyn PhysicalExpr>)
+ } else {
+ None
+ };
+
+ let filter_expr = match (maybe_bounds_expr, maybe_hash_eval_expr) {
+ (None, Some(expr)) | (Some(expr), None) => expr,
+ // Bounds collection and hash collection are current mutually exclusive, so this branch should not be
+ // taken at the moment. However, if they are ever enabled together, this is how we would combine them.
+ (Some(bounds_expr), Some(hash_eval_expr)) => Arc::new(BinaryExpr::new(
+ bounds_expr,
+ Operator::And,
+ Arc::clone(&hash_eval_expr),
+ )),
+ _ => return Ok(()),
+ };
+
+ self.dynamic_filter.update(filter_expr)?;
+ }
+
+ Ok(())
+ }
+}
+
+impl fmt::Debug for SharedBuildAccumulator {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ write!(f, "SharedBuildAccumulator")
+ }
+}
+
+/// A [`PhysicalExpr`] that evaluates to a boolean array indicating which rows in a batch
+/// have hashes that exist in a given set of hashes.
+///
+/// This is currently used to implement hash-based dynamic filters in hash joins. That is,
+/// this expression can be pushed down to the probe side of a hash join to filter out rows
+/// that do not have a matching hash in the build side, allowing the scan to emit only the rows
+/// that are guaranteed to have at least one match.
+struct HashComparePhysicalExpr {
+ /// Expressions that will be evaluated to compute hashes for filtering
+ exprs: Vec<Arc<dyn PhysicalExpr>>,
+ /// Hashes to filter against
+ hashes: Arc<NoHashSet<u64>>,
+ /// Random state for hash computation
+ random_state: &'static RandomState,
+ /// Memory reservation used to track the memory used by `hashes`
+ reservation: MemoryReservation,
+}
+
+impl HashComparePhysicalExpr {
+ pub fn new(
+ exprs: Vec<Arc<dyn PhysicalExpr>>,
+ hashes: Arc<NoHashSet<u64>>,
+ random_state: &'static RandomState,
+ reservation: MemoryReservation,
+ ) -> Self {
+ Self {
+ exprs,
+ hashes,
+ random_state,
+ reservation,
+ }
+ }
+}
+
+impl fmt::Debug for HashComparePhysicalExpr {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ write!(f, "HashComparePhysicalExpr [ {:?} ]", self.exprs)
+ }
+}
+
+impl Hash for HashComparePhysicalExpr {
+ fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
+ self.exprs.hash(state);
+ }
+}
+
+impl PartialEq for HashComparePhysicalExpr {
+ fn eq(&self, other: &Self) -> bool {
+ // TODO: We limit comparison to uphold the equality property w.r.t `Hash`.
+ // However, this means we may consider two expressions equal when they are actually not.
+ // Since this is currently just used internally for dynamic filters, this may be acceptable
+ // for now. If this expression is ever exposed more broadly, we should revisit this.
+ self.exprs == other.exprs
+ }
+}
+
+impl Eq for HashComparePhysicalExpr {}
+
+impl fmt::Display for HashComparePhysicalExpr {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ let exprs = self
+ .exprs
+ .iter()
+ .map(|e| e.to_string())
+ .collect::<Vec<_>>()
+ .join(", ");
+ write!(f, "HashComparePhysicalExpr [ {exprs} ]")
+ }
+}
+
+impl PhysicalExpr for HashComparePhysicalExpr {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
+ self.exprs.iter().collect()
+ }
+
+ fn with_new_children(
+ self: Arc<Self>,
+ children: Vec<Arc<dyn PhysicalExpr>>,
+ ) -> Result<Arc<dyn PhysicalExpr>> {
+ Ok(Arc::new(Self {
+ exprs: children,
+ hashes: Arc::clone(&self.hashes),
+ random_state: self.random_state,
+ reservation: self.reservation.new_empty(),
+ }))
+ }
+
+ fn data_type(&self, _input_schema: &Schema) -> Result<DataType> {
+ Ok(DataType::Boolean)
+ }
+
+ fn nullable(&self, _input_schema: &Schema) -> Result<bool> {
+ Ok(false)
+ }
+
+ fn evaluate(
+ &self,
+ batch: &arrow::record_batch::RecordBatch,
+ ) -> Result<ColumnarValue> {
+ let num_rows = batch.num_rows();
+
+ let expr_values = self
+ .exprs
+ .iter()
+ .map(|col| col.evaluate(batch)?.into_array(num_rows))
+ .collect::<Result<Vec<_>>>()?;
+
+ // Compute hashes for each row based on the evaluated expressions
+ let mut hashes_buffer = vec![0; num_rows];
+ create_hashes(&expr_values, self.random_state, &mut hashes_buffer)?;
+
+ // Create a boolean array where each position indicates if the corresponding hash is in the set of known hashes
+ let mut buf = MutableBuffer::from_len_zeroed(bit_util::ceil(num_rows, 8));
Review Comment:
Should be able to use `MutableBuffer::collect_bool` instead of setting bits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2460976149
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
@LiaCastaneda do you mean because of hash collisions?
@Dandandan my understanding is that the better performance came from not sequentially checking each hash table for each partition. But I haven't seen that code, I'd guess it could be optimized e.g. using a `CASE hash(col) % num_partitions WHEN 0 THEN <check table 0> WHEN 1 <check table 1> ...`. Hence my desire to have a bakeoff similar to https://github.com/apache/datafusion/pull/17529#issuecomment-3379650299
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2413960610
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
That makes sense, out plan for setting bounds is to use some available statistics we have ( like row count), and set the `enable_dynamic...` option based on that, so its even before logical and physical DataFusion planning. Would it be possible to do something similar in DataFusion itself while running the optimizer rule?
Although the option you mention is more straightforward, are you suggesting that we set a configuration option for maximum memory consumption and, when building the hashes, stop producing them and free memory if we exceed that limit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#issuecomment-3423727231
To address the memory concern, have we consider switching to a bloom filter at a certain size?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2354338401
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
I worry that the extra memory cost is prohibitively expensive: there are going to be queries that ran just fine previously but now OOM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2353830615
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
I found using a `HashSet` here to yield better performance than using a `Vec<Arc<dyn JoinHashMapType>>`. Though, it does of course result in extra allocations.
My guess is that its primarily due to that `Vec<Arc<dyn JoinHashMapType>>` results in more indirection + scattered memory accesses which likely means worse cache locality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#issuecomment-3300583673
@alamb Would you be able to kick off benchmarks 🙏🏾 ? Specifically TPC-H against parquet files
Should want the following configuration options set:
```
DATAFUSION_EXECUTION_HASH_JOIN_SIDEWAYS_HASH_PASSING=true
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS=true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#issuecomment-3838532799
> @rkrishn7 I think this work has been superseded and this PR now has many conflicts - should we close it? Sorry we never got to include your work...
@adriangb Yes! Let's close it. And no worries at all - thank you for picking up the work here! Admittedly I got a bit busy with some other things and saw you were driving this, so much appreciated 🙌🏾
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#issuecomment-3445109252
Thanks for the reviews and testing @adriangb @LiaCastaneda @Dandandan
Apologies, I haven't had time to look at this the past couple weeks. I'll be able to give this more attention next week when I'm back home.
Still very excited about this work and looking forward to addressing everyone's feedback/suggestions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2464881216
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
> @LiaCastaneda do you mean because of hash collisions?
I was thinking because localized lookups would be more efficient. iiuc, probe partition 0 should only check build partition 0's hash table, and so on. The problem is that on the probe side, in the `evaluate()` function (when the dynamic filter runs), we don't have information about which partition the batch belongs to.
I wonder if we can compute this "routing" using the `RepartitionExec's` hashing to figure out the partition, then use the join's random state for the actual hash lookup. I tried this two-hash approach branching off this PR and have something [here](https://github.com/LiaCastaneda/datafusion/commit/f131f768a41b30ace94a091b681a22a589361624#diff-4d7e84ce1bf99fb0e20194db32f4c9ad283815196973d7601c6b36a41392e03a) that returns correct results while using less memory in most cases (e.g., Q9 and Q18 save ~600MB each). @rkrishn7 feel free to take it if it helps or if it even makes sense :) I was just trying things out here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2464108730
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
My proposal I think is slightly different
* for > 1 table, we can use the current hashset logic (bus use to `JoinHashMapType` trait or something similar to encode the "contains"). This would still be a new copy.
* For 1 table, use the hashmap as-is (i.e. without overhead).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2480254592
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
Ran your commit locally @LiaCastaneda and can confirm I see similar results as well! Strange - I thought I was doing essentially the same thing in my comparisons from a while ago 🤔 . Only difference then is I was running on my beefier Linux computer which I don't have at the moment, but I definitely could have just missed something.
Anyhow, since results look similar I propose we move forward with the approach of reusing the hash table(s)! I can apply your patch to this branch over the weekend if that sounds good. And thanks again for putting that up 🙌🏾
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2451923599
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
If it is mainly from having `Vec<Arc<dyn JoinHashMapType>>` (i.e. vec of tables) and doing multiple lookups, maybe we can consider `Arc<dyn JoinHashMapType>` instead?
1. For pushing down 1 table, just keep the original table
2. For >1 tables, merge the tables to one `JoinHashMapType` (with offset set to 0).
This will avoid the creation overhead for 1 table. If we have benchmarks it works better for a higher number of tables (e.g. 2 or 3) we can do so to, to reduce the overhead of creating a new table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2451891270
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
I don't fully understand yet why `HashSet` would yield better performance? The hashmap is not much more than a hashset of hashes (to first offset) itself, so shouldn't be much slower to test if the table contains the hash?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2356297965
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
Since this is opt-in, hopefully it's not as much of an issue? I've mentioned this in the config documentation:
```
/// When set to true, hash joins will allow passing hashes from the build
/// side to the right side of the join. This can be useful to prune rows early on,
/// but may consume more memory.
```
In general though I agree that we shouldn't _need_ extra allocations here. It gets a bit tricky though because even if we combine all the hash tables from each build partition into a single, shareable table - each stream probe partition needs to be able to validate that the lookup is localized to its partition. Otherwise we'll see duplicate / incorrect results I believe.
Though, it may just take some tweaking to the existing data structure. Haven't thought about it enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2464886755
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
> @LiaCastaneda do you mean because of hash collisions?
I was thinking because localized lookups would be more efficient. iiuc, probe partition 0 should only check build partition 0's hash table, and so on. The problem is that on the probe side, in the `evaluate()` function (when the dynamic filter runs), we don't have information about which partition the batch belongs to.
I wonder if we can compute this "routing" using the `RepartitionExec` hashing to figure out the partition, then use the join's hashing for the actual hash lookup. I tried this two hash approach branching off this PR and have something [here](https://github.com/LiaCastaneda/datafusion/commit/f131f768a41b30ace94a091b681a22a589361624#diff-4d7e84ce1bf99fb0e20194db32f4c9ad283815196973d7601c6b36a41392e03a), I think it returns correct results while using less memory in most cases (e.g., Q9 and Q18 save ~600MB each). @rkrishn7 feel free to take it if it helps or if it even makes sense :) I was just trying things out here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2411406442
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
Hi @LiaCastaneda - that would be great! And apologies for the delayed response.
I am curious to see how gains diminish as the build side grows larger. I'm not sure if it is that drastic or if there are still gains to be seen for very large build sides. To me the biggest downside is memory usage. e.g. a build side with 1 billion hashes would be ~8GB of additional memory.
Currently there are no checks to determine whether it is worth it to build the hash set and push it down. Though, that could be done via separate configuration depending on the upper bound of memory consumption a consumer is comfortable with for this feature 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2451923599
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
If it is mainly from having `Vec<Arc<dyn JoinHashMapType>>` (i.e. vec of tables) and doing multiple lookups, maybe we can consider `Arc<dyn JoinHashMapType>` instead?
1. For pushing down 1 table, just keep the original table
2. For >1 tables, merge the tables to one `JoinHashMapType` (with a new`JoinHashMapType` containing only hashes).
This will avoid the creation overhead for 1 table. If we have benchmarks it works better for a higher number of tables (e.g. 2 or 3) we can do so to, to reduce the overhead of creating a new table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2451923599
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
If it is mainly from having `Vec<Arc<dyn JoinHashMapType>>` (i.e. vec of tables) and doing multiple lookups, maybe we can consider `Arc<dyn JoinHashMapType>` instead?
1. For pushing down 1 table, just keep the original table
2. For >1 tables, merge the tables to one `JoinHashMapType` (with offset set to 0).
This will avoid the creation overhead for 1 table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2462236407
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
> If it is mainly from having Vec<Arc<dyn JoinHashMapType>> (i.e. vec of tables) and doing multiple lookups, maybe we can consider Arc<dyn JoinHashMapType> instead?
@Dandandan Yeah that's my guess. I imagine that unless we continue to hit the same index in the vector, we would see a performance hit.
> For >1 tables, merge the tables to one JoinHashMapType (with a newJoinHashMapType containing only hashes).
I was thinking about this and I found it hard in practice to merge the maps. Since probe sides operate on partitions, we can't share a global hash map that contains indices for all partitions on the build side. If we did want to share the hash maps, I think we'd have to:
- Somehow encode partition information in the existing structures that implement `JoinHashMapType`. This is so probe side lookups yield indices for only their respective partitions.
- Continue to wait until build sides are processed before moving on to probe side, to remain lock-free. Though IIRC @adriangb had some work to remove the barrier synchronization there.
My memory is a bit foggy here so I could be missing something
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#issuecomment-3304022192
> > @alamb Would you be able to kick off benchmarks 🙏🏾 ? Specifically TPC-H against parquet files
> > Should want the following configuration options set:
> > ```
> > DATAFUSION_EXECUTION_HASH_JOIN_SIDEWAYS_HASH_PASSING=true
> > DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS=true
> > ```
>
> I am not quite sure how to set these options in the benchmarks...
Ah okay then we probably won't see any changes since these need to be enabled for the changes here to take effect. Posting a comparison I did locally:
```
Comparing main and feat_pushdown-hashes-hashjoinexec
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ main ┃ feat_pushdown-hashes-hashjoinexec ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 58.88 ms │ 57.03 ms │ no change │
│ QQuery 2 │ 53.69 ms │ 30.18 ms │ +1.78x faster │
│ QQuery 3 │ 66.45 ms │ 68.39 ms │ no change │
│ QQuery 4 │ 41.55 ms │ 41.23 ms │ no change │
│ QQuery 5 │ 108.33 ms │ 68.41 ms │ +1.58x faster │
│ QQuery 6 │ 28.36 ms │ 26.23 ms │ +1.08x faster │
│ QQuery 7 │ 101.16 ms │ 56.93 ms │ +1.78x faster │
│ QQuery 8 │ 111.82 ms │ 78.32 ms │ +1.43x faster │
│ QQuery 9 │ 126.30 ms │ 127.26 ms │ no change │
│ QQuery 10 │ 94.72 ms │ 64.64 ms │ +1.47x faster │
│ QQuery 11 │ 36.03 ms │ 17.83 ms │ +2.02x faster │
│ QQuery 12 │ 69.64 ms │ 58.65 ms │ +1.19x faster │
│ QQuery 13 │ 34.31 ms │ 33.93 ms │ no change │
│ QQuery 14 │ 43.37 ms │ 41.74 ms │ no change │
│ QQuery 15 │ 49.23 ms │ 48.76 ms │ no change │
│ QQuery 16 │ 35.28 ms │ 30.01 ms │ +1.18x faster │
│ QQuery 17 │ 125.91 ms │ 112.71 ms │ +1.12x faster │
│ QQuery 18 │ 160.70 ms │ 203.86 ms │ 1.27x slower │
│ QQuery 19 │ 81.69 ms │ 68.15 ms │ +1.20x faster │
│ QQuery 20 │ 47.11 ms │ 47.74 ms │ no change │
│ QQuery 21 │ 149.61 ms │ 113.31 ms │ +1.32x faster │
│ QQuery 22 │ 18.61 ms │ 17.81 ms │ no change │
└──────────────┴───────────┴───────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (main) │ 1642.74ms │
│ Total Time (feat_pushdown-hashes-hashjoinexec) │ 1413.10ms │
│ Average Time (main) │ 74.67ms │
│ Average Time (feat_pushdown-hashes-hashjoinexec) │ 64.23ms │
│ Queries Faster │ 12 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 9 │
│ Queries with Failure │ 0 │
└──────────────────────────────────────────────────┴───────────┘
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#issuecomment-3301353213
Very cool!
Incidentally we were just discussing today with @gabotechs and @robtandy how to make HashJoin dynamic filter pushdown more compatible with distributed datafusion and how to eliminate the latency associated with waiting until we have the full build side to create filters.
One idea that came up was to push something like:
`(hash(join_key_col) % parts != 0) or join_key_col >= 1 and join_key_col <= 2` where `0` is the partition number, `1` is the min val for `join_key_col` in that partition and `2` is the max val
We can push these down "as they are ready" and then once the all partitions are ready we can simplify this to our current `join_key_col >= 1 and join_key_col <= 2`.
I bring this up because the main sticking point with that approach is that we add third computation of the hash to the existing two (in the hash join and repartition/shuffle), which might have a performance impact.
That led us to file https://github.com/apache/datafusion/issues/17599 which is a broader issue that DataFusion has.
But for this PR the big question in my mind is going to be: is the cost of the extra evaluation of the hash worth it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 closed pull request #17529: feat: Push down hashes to probe side in HashJoinExec
URL: https://github.com/apache/datafusion/pull/17529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#issuecomment-3838645017
Totally understand! Just sad to see great work never get merged. I’d love to have you back for more contributions if you find time. This work (producing filters from joins) in particular still needs implementing for other join types, and it would be great to allow filter pushdown through more join types as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2354318648
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
+ /// Memory reservation tracking the memory used by `left_hashes`
+ reservation: MemoryReservation,
+}
+
+impl SharedBuildAccumulator {
+ /// Creates a new [SharedBuildAccumulator] configured for the given partition mode
Review Comment:
```suggestion
/// Creates a new [`SharedBuildAccumulator`] configured for the given partition mode
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#issuecomment-3379650299
@rkrishn7 sorry if I haven't looped back here. I went on a bit of a tangent exploring https://github.com/apache/datafusion/pull/17632 and then had some vacation and a team offsite. This is overall very exciting work that I think will help a lot of people.
My main concern with this change is the overhead and making a CPU / memory tradeoff decision for users. I think we might be able to ship it as an experimental thing with the feature flag defaulting to false as you've done in this PR but long term I worry that an [extra 8GB or RAM consumed](https://github.com/apache/datafusion/pull/17529#discussion_r2411406442) might be too much. Do you have any numbers on how fast and how much RAM these 3 different scenarios use for some queries? I don't mean to ask you to run them all but I do remember you mentioning you have already.
1. No hashes pushdown.
2. Build a single hash table (this PR).
3. Push down the existing hash tables w/ a CASE statement for which hash table to check based on the same partitioning as RepartitionExec.
My hypothesis is that the table will look something like this:
| | (1) | (2) | (3) |
|----------|------|------|-----|
| Runtime | 300s | 25s | 30s |
| Peak mem | 8GB | 16GB | 8GB |
*If* that were the case, which is just a guess at this point, making a query 10x faster with no extra memory use is easy to justify, everyone wants that! Choosing to make some queries 17% faster for 100% more memory use is harder to justify. If the performance difference is larger and we think it is justified in some cases maybe we can at least try to reserve the extra memory and fall back to re-using the existing hash tables?
I also think it's worth thinking about integrating your suggestion from our conversation to use an `IN LIST` expression because it's even integrated into bloom filter pruning and predicate pruning. I see basically no reason to not do that for small build side result sets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2401473105
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
👋 I'd be happy to test this in our service, and see memory implications in a prod environment. I guess for any query that uses this kind of filter, the Hash table shouldn't be big? otherwise this kind of filters wont be that worth it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#issuecomment-3429689554
👋 I was playing around with this feature today, here are some results for sf1 and sf10, claude did these nice summaries
```
| Query | Baseline Time | Hash Pushdown Time | Time Change | Baseline Memory | Hash Memory | Memory Change |
|-------|---------------|--------------------|----------------|-----------------|-------------|---------------|
| Q1 | 55.80 ms | 59.15 ms | 1.06x slower | 378.9 MB | 349.0 MB | -29.9 MB |
| Q2 | 32.31 ms | 21.93 ms | 1.47x faster | 243.4 MB | 223.8 MB | -19.6 MB |
| Q3 | 50.02 ms | 43.97 ms | 1.14x faster | 465.0 MB | 438.5 MB | -26.5 MB |
| Q4 | 34.93 ms | 36.87 ms | 1.06x slower | 639.0 MB | 656.9 MB | +17.9 MB |
| Q5 | 58.00 ms | 52.43 ms | 1.11x faster | 458.2 MB | 453.5 MB | -4.7 MB |
| Q6 | 24.32 ms | 26.13 ms | 1.07x slower | 430.4 MB | 422.9 MB | -7.5 MB |
| Q7 | 61.62 ms | 39.10 ms | 1.58x faster | 522.3 MB | 494.2 MB | -28.1 MB |
| Q8 | 54.45 ms | 47.40 ms | 1.15x faster | 597.3 MB | 576.9 MB | -20.4 MB |
| Q9 | 61.88 ms | 78.72 ms | 1.27x slower | 649.0 MB | 729.8 MB | +80.8 MB |
| Q10 | 61.38 ms | 43.34 ms | 1.42x faster | 493.4 MB | 457.4 MB | -36.0 MB |
| Q11 | 21.78 ms | 15.00 ms | 1.45x faster | 213.7 MB | 173.5 MB | -40.2 MB |
| Q12 | 56.02 ms | 53.81 ms | 1.04x faster | 507.0 MB | 480.0 MB | -27.0 MB |
| Q13 | 27.48 ms | 26.66 ms | no change | 386.6 MB | 411.0 MB | +24.4 MB |
| Q14 | 25.54 ms | 29.37 ms | 1.15x slower | 410.0 MB | 406.6 MB | -3.4 MB |
| Q15 | 41.22 ms | 40.93 ms | no change | 420.3 MB | 420.8 MB | +0.5 MB |
| Q16 | 18.24 ms | 16.11 ms | 1.13x faster | 196.8 MB | 193.5 MB | -3.3 MB |
| Q17 | 68.96 ms | 54.58 ms | 1.26x faster | 739.0 MB | 702.2 MB | -36.8 MB |
| Q18 | 78.27 ms | 99.72 ms | 1.27x slower | 646.4 MB | 763.2 MB | +116.8 MB |
| Q19 | 40.37 ms | 37.09 ms | 1.09x faster | 546.1 MB | 555.2 MB | +9.1 MB |
| Q20 | 34.68 ms | 35.96 ms | no change | 466.8 MB | 477.9 MB | +11.1 MB |
| Q21 | 102.68 ms | 87.24 ms | 1.18x faster | 714.5 MB | 602.4 MB | -112.1 MB |
| Q22 | 12.09 ms | 13.67 ms | 1.13x slower | 205.6 MB | 205.6 MB | 0.0 MB |
```
SF10
```
| Query | Baseline Time | Hash Pushdown Time | Time Change | Baseline Memory | Hash Memory | Memory Change |
|-------|---------------|--------------------| -----------------|-----------------|-------------|---------------|
| Q1 | 472.58 ms | 502.66 ms | 1.06x slower | 835.4 MB | 844.5 MB | +9.1 MB |
| Q2 | 131.42 ms | 109.07 ms | 1.21x faster | 734.1 MB | 766.8 MB | +32.7 MB |
| Q3 | 428.75 ms | 287.98 ms | 1.49x faster | 1.3 GB | 1.1 GB | -194.2 MB |
| Q4 | 327.30 ms | 341.23 ms | 1.04x slower | 3.2 GB | 3.4 GB | +200 MB |
| Q5 | 430.55 ms | 409.65 ms | 1.05x faster | 1.3 GB | 1.3 GB | +75 MB |
| Q6 | 172.13 ms | 169.20 ms | no change | 1.0 GB | 1.0 GB | +8 MB |
| Q7 | 423.89 ms | 284.72 ms | 1.49x faster | 1.3 GB | 1.3 GB | -4.1 MB |
| Q8 | 559.76 ms | 454.55 ms | 1.23x faster | 1.8 GB | 1.9 GB | +45.8 MB |
| Q9 | 697.39 ms | 835.67 ms | 1.20x slower | 2.7 GB | 3.4 GB | +700 MB |
| Q10 | 533.91 ms | 363.03 ms | 1.47x faster | 1.8 GB | 1.6 GB | -138.1 MB |
| Q11 | 90.07 ms | 63.19 ms | 1.43x faster | 514.3 MB | 564.8 MB | +50.5 MB |
| Q12 | 469.67 ms | 451.50 ms | 1.04x faster | 1.3 GB | 1.4 GB | +50.1 MB |
| Q13 | 306.60 ms | 284.17 ms | 1.08x faster | 2.8 GB | 2.3 GB | -500 MB |
| Q14 | 220.53 ms | 225.02 ms | no change | 1.1 GB | 1.2 GB | +111.3 MB |
| Q15 | 321.28 ms | 315.00 ms | no change | 969.6 MB | 1.1 GB | +95.2 MB |
| Q16 | 81.82 ms | 79.00 ms | no change | 766.1 MB | 794.2 MB | +28.1 MB |
| Q17 | 694.52 ms | 512.22 ms | 1.36x faster | 1.2 GB | 1.3 GB | +15.4 MB |
| Q18 | 904.41 ms | 1290.26 ms | 1.43x slower | 4.4 GB | 5.1 GB | +700 MB |
| Q19 | 330.76 ms | 273.10 ms | 1.21x faster | 1.6 GB | 1.6 GB | +26.7 MB |
| Q20 | 290.40 ms | 282.53 ms | no change | 1.7 GB | 1.6 GB | -59.3 MB |
| Q21 | 947.20 ms | 789.62 ms | 1.20x faster | 2.0 GB | 2.0 GB | +28.7 MB |
| Q22 | 89.73 ms | 88.49 ms | no change | 615.0 MB | 615.1 MB | +0.1 MB |
```
Memory measurements are from `Peak RSS` which iiuc is the peak memory used by the process, not sure if this is the best way to test memory. Also, I'm not sure how we can test huge build sides with ~1B hashes, which makes me wonder what would be a good way to track how many hashes are generated per query, maybe include it as a metric of the join.
In any case, with some manual logging for Q18 (which appears to be the heaviest join), I'm seeing 1.5M hashes for SF1 and 15M hashes for SF10. The correlation between more data and more memory is clear, but IMO, if there's a way to measure the total number of hashes across partitions, could we opt out of the feature during runtime? and allow the user to configure this based on their available resources/ pod size
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "rkrishn7 (via GitHub)" <gi...@apache.org>.
rkrishn7 commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2356322889
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
Ah yeah that's essentially the same as what I was referring to earlier with using a `Vec<Arc<dyn JoinHashMapType>>` (sorry probably should've clarified more).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#issuecomment-3304000145
🤖 `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running
Linux aal-dev 6.14.0-1014-gcp #15~24.04.1-Ubuntu SMP Fri Jul 25 23:26:08 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing feat/pushdown-hashes-hashjoinexec (76e0c0a7b295a2bddab39e6097607bc67b20eca9) to 0181c7906b47f9b17fbde92bf30b2d939a80230c [diff](https://github.com/apache/datafusion/compare/0181c7906b47f9b17fbde92bf30b2d939a80230c..76e0c0a7b295a2bddab39e6097607bc67b20eca9) using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#issuecomment-3304001492
> @alamb Would you be able to kick off benchmarks 🙏🏾 ? Specifically TPC-H against parquet files
>
> Should want the following configuration options set:
>
> ```
> DATAFUSION_EXECUTION_HASH_JOIN_SIDEWAYS_HASH_PASSING=true
> DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS=true
> ```
I am not quite sure how to set these options in the benchmarks...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Push down hashes to probe side in HashJoinExec [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #17529:
URL: https://github.com/apache/datafusion/pull/17529#discussion_r2464881216
##########
datafusion/physical-plan/src/joins/hash_join/information_passing.rs:
##########
@@ -0,0 +1,612 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Utilities for passing information from the build side of a hash join to the probe side.
+//! This is also known as "sideways information passing", and enables optimizations that take
+//! of the smaller build side to reduce data processed on the larger probe side.
+//!
+//! As an example, let's consider TPC-H Query 17:
+//!
+//! ```sql
+//! select
+//! sum(l_extendedprice) / 7.0 as avg_yearly
+//! from
+//! lineitem,
+//! part
+//! where
+//! p_partkey = l_partkey
+//! and p_brand = 'Brand#23'
+//! and p_container = 'MED BOX'
+//! and l_quantity < (
+//! select
+//! 0.2 * avg(l_quantity)
+//! from
+//! lineitem
+//! where
+//! l_partkey = p_partkey
+//! );
+//! ```
+//! The join portion of the query should look something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │ This scan decodes 60M values of l_quantity and l_extendedprice,
+//! ▼ ▼ even though all but 2044 are filtered by the join!
+//! ┌──────────────────┐ ┌─────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ │
+//! │projection: │ │projection: │ │
+//! │ p_partkey │ │ l_quantity, │ │
+//! │filters: │ │ l_extendedprice, │◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ p_brand = .. │ │ l_partkey │
+//! │ p_container = ..│ │filters: │
+//! │ │ │ NONE │
+//! └──────────────────┘ └─────────────────────┘
+//! ```
+//!
+//! The key observation is that the scan of `lineitem` produces 60 million rows, but only 2044 of them
+//! will pass the join condition. If we can push down a filter to the scan of `lineitem` that only limits
+//! the number of rows produced, we can avoid a lot of unnecessary work.
+//!
+//! Given that in a hash join, we fully process the build side (in this case, `part`) before scanning partitions
+//! of the probe side (`lineitem`), we can collect information from the build side that helps us construct filters
+//! for the probe side. This allows us to transform the above plan into something like this:
+//!
+//! ```text
+//! │
+//! │
+//! 2044 Rows │
+//! │
+//! ▼
+//! ┌────────────────┐
+//! │ HashJoin │
+//! │ p_partkey = │
+//! │ l_partkey │
+//! └──┬─────────┬───┘
+//! 2M Rows │ │ 60M Rows
+//! │ │
+//! │ │
+//! ┌────────┘ └─────────┐
+//! │ │
+//! ▼ ▼
+//! ┌──────────────────┐ ┌──────────────────────────────┐
+//! │Scan: part │ │Scan: lineitem │ Now, the scan contains a filter that takes into account
+//! │projection: │ │projection: │ min/max bounds from the left side. This enables scans that
+//! │ p_partkey │ │ l_quantity, │ take advantage of late materialization to avoid decoding
+//! │filters: │ │ l_extendedprice, │ l_quantity and l_extendedprice for rows that do not match.
+//! │ p_brand = .. │ │ l_partkey │ │
+//! │ p_container = ..│ │filters: │ │
+//! └──────────────────┘ │ l_partkey >= min(p_partkey) │ │
+//! │ and │ ◀─ ─ ─ ─ ─ ─ ─ ─ ─
+//! │ l_partkey <= max(p_partkey) │
+//! │ and │
+//! │ ... │
+//! └──────────────────────────────┘
+//! ```
+//!
+//! Dynamic filters are expressions that allow us to pass information sideways in a query plan. They essentially start out
+//! as dummy filters that are always true (resulting in no selectivity), and may be updated later during query execution.
+//! In the case of a hash join, we update dynamic filters after fully processing the build side, and before scanning the probe side.
+//!
+//! References:
+//! - <https://github.com/apache/datafusion/issues/7955>
+//! - <https://datafusion.apache.org/blog/2025/09/10/dynamic-filters/>
+
+use std::{any::Any, fmt, hash::Hash, sync::Arc};
+
+use crate::joins::utils::JoinHashMapType;
+use crate::joins::utils::NoHashHasher;
+use crate::joins::utils::NoHashSet;
+use crate::joins::PartitionMode;
+use crate::ExecutionPlan;
+use crate::ExecutionPlanProperties;
+
+use ahash::RandomState;
+use arrow::{
+ array::BooleanArray,
+ buffer::MutableBuffer,
+ datatypes::{DataType, Schema},
+ util::bit_util,
+};
+use datafusion_common::utils::memory::estimate_memory_size;
+use datafusion_common::HashSet;
+use datafusion_common::{hash_utils::create_hashes, Result, ScalarValue};
+use datafusion_execution::memory_pool::MemoryReservation;
+use datafusion_expr::{ColumnarValue, Operator};
+use datafusion_physical_expr::expressions::{lit, BinaryExpr, DynamicFilterPhysicalExpr};
+use datafusion_physical_expr::PhysicalExpr;
+
+use itertools::Itertools;
+use parking_lot::Mutex;
+use tokio::sync::Barrier;
+
+/// Represents the minimum and maximum values for a specific column.
+/// Used in dynamic filter pushdown to establish value boundaries.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) struct ColumnBounds {
+ /// The minimum value observed for this column
+ min: ScalarValue,
+ /// The maximum value observed for this column
+ max: ScalarValue,
+}
+
+impl ColumnBounds {
+ pub(crate) fn new(min: ScalarValue, max: ScalarValue) -> Self {
+ Self { min, max }
+ }
+}
+
+/// Represents the bounds for all join key columns from a single partition.
+/// This contains the min/max values computed from one partition's build-side data.
+#[derive(Debug, Clone)]
+pub(crate) struct PartitionBounds {
+ /// Partition identifier for debugging and determinism (not strictly necessary)
+ partition: usize,
+ /// Min/max bounds for each join key column in this partition.
+ /// Index corresponds to the join key expression index.
+ column_bounds: Vec<ColumnBounds>,
+}
+
+impl PartitionBounds {
+ pub(crate) fn new(partition: usize, column_bounds: Vec<ColumnBounds>) -> Self {
+ Self {
+ partition,
+ column_bounds,
+ }
+ }
+
+ pub(crate) fn len(&self) -> usize {
+ self.column_bounds.len()
+ }
+
+ pub(crate) fn get_column_bounds(&self, index: usize) -> Option<&ColumnBounds> {
+ self.column_bounds.get(index)
+ }
+}
+
+/// Coordinates information collected from the build side of the join, across multiple partitions
+///
+/// This structure ensures that dynamic filters are built with complete information from all
+/// relevant partitions before being applied to probe-side scans. Incomplete filters would
+/// incorrectly eliminate valid join results.
+///
+/// ## Synchronization Strategy
+///
+/// 1. Each partition relays values computed from its build-side data (e.g. min/max bounds, available hashes)
+/// 2. The per-partition information is accumulated in shared state
+/// 3. A [`Barrier`] is used to track reporters. Once the last partition reports, the information is merged and the filter is updated exactly once
+///
+/// ## Partition Counting
+///
+/// The `total_partitions` count represents how many times `collect_build_side` will be called:
+///
+/// - **CollectLeft**: Number of output partitions (each accesses shared build data)
+/// - **Partitioned**: Number of input partitions (each builds independently)
+///
+/// ## Thread Safety
+///
+/// All fields use a single mutex to ensure correct coordination between concurrent
+/// partition executions.
+pub(crate) struct SharedBuildAccumulator {
+ /// Shared state protected by a single mutex to avoid ordering concerns
+ inner: Mutex<SharedBuildState>,
+ /// Synchronization barrier to track when all partitions have reported
+ barrier: Barrier,
+ /// Dynamic filter for pushdown to probe side
+ dynamic_filter: Arc<DynamicFilterPhysicalExpr>,
+ /// Right side join expressions needed for creating filter bounds
+ on_right: Vec<Arc<dyn PhysicalExpr>>,
+ /// Random state used for hash computation
+ random_state: &'static RandomState,
+}
+
+/// State protected by SharedBuildAccumulator's mutex
+struct SharedBuildState {
+ /// Bounds from completed partitions.
+ /// Each element represents the column bounds computed by one partition.
+ bounds: Vec<PartitionBounds>,
+ /// Hashes from the left (build) side, if enabled
+ left_hashes: NoHashSet<u64>,
Review Comment:
> @LiaCastaneda do you mean because of hash collisions?
I was thinking because localized lookups would be more efficient. iiuc, probe partition 0 should only check build partition 0's hash table, and so on. The problem is that on the probe side, in the `evaluate()` function (when the dynamic filter runs), we don't have information about which partition the batch belongs to.
I wonder if we can compute this "routing" using the `RepartitionExec's` hashing to figure out the partition, then use the join's random state for the actual hash lookup. I tried this two-hash approach branching off this PR and have something [here](https://github.com/LiaCastaneda/datafusion/commit/f131f768a41b30ace94a091b681a22a589361624#diff-4d7e84ce1bf99fb0e20194db32f4c9ad283815196973d7601c6b36a41392e03a) that returns correct results while using less memory in most cases (e.g., Q9 and Q18 save ~600MB each). @rkrishn7 feel free to take it if it helps or if it even makes sense :) I was just trying things out here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org