[GitHub] [arrow-datafusion] korowa commented on a diff in pull request #2591: Optional filter in `JOIN ON` clause

[GitBox <gi...@apache.org>]

korowa commented on code in PR #2591:
URL: https://github.com/apache/arrow-datafusion/pull/2591#discussion_r879834683


##########
datafusion/core/src/physical_plan/hash_join.rs:
##########
@@ -791,6 +826,109 @@ fn build_join_indexes(
     }
 }
 
+fn apply_join_filter(
+    left: &RecordBatch,
+    right: &RecordBatch,
+    join_type: JoinType,
+    left_indices: UInt64Array,
+    right_indices: UInt32Array,
+    filter: &JoinFilter,
+) -> Result<(UInt64Array, UInt32Array)> {
+    if left_indices.is_empty() {
+        return Ok((left_indices, right_indices));
+    };
+
+    let (intermediate_batch, _) = build_batch_from_indices(
+        filter.schema(),
+        left,
+        right,
+        PrimitiveArray::from(left_indices.data().clone()),
+        PrimitiveArray::from(right_indices.data().clone()),
+        filter.column_indices(),
+    )?;
+
+    match join_type {
+        JoinType::Inner | JoinType::Left => {
+            // For both INNER and LEFT joins, input arrays contains only indices for matched data.
+            // Due to this fact it's correct to simply apply filter to intermediate batch and return
+            // indices for left/right rows satisfying filter predicate
+            let filter_result = filter
+                .expression()
+                .evaluate(&intermediate_batch)?
+                .into_array(intermediate_batch.num_rows());
+            let mask = as_boolean_array(&filter_result);
+
+            let left_filtered = PrimitiveArray::<UInt64Type>::from(
+                compute::filter(&left_indices, mask)?.data().clone(),

Review Comment:
   Yes, at physical level filter always applied after actual join operation.
   
   And you're right - there are cases when it's fine to filter inputs before join step - but here it's responsibility of logical planner / optimizer - they both are able to check if predicate (or its part) could be pushed before join - closer to scan.
   
   So, when it comes to physical join, logical plan supposed to contain only `t1.field < t2.field`-like predicates which could not be applied before join



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org