You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/05/23 15:49:16 UTC

[GitHub] [arrow-datafusion] Ted-Jiang commented on a diff in pull request #2591: Optional filter in `JOIN ON` clause

Ted-Jiang commented on code in PR #2591:
URL: https://github.com/apache/arrow-datafusion/pull/2591#discussion_r879616616


##########
datafusion/core/src/physical_plan/hash_join.rs:
##########
@@ -791,6 +826,109 @@ fn build_join_indexes(
     }
 }
 
+fn apply_join_filter(
+    left: &RecordBatch,
+    right: &RecordBatch,
+    join_type: JoinType,
+    left_indices: UInt64Array,
+    right_indices: UInt32Array,
+    filter: &JoinFilter,
+) -> Result<(UInt64Array, UInt32Array)> {
+    if left_indices.is_empty() {
+        return Ok((left_indices, right_indices));
+    };
+
+    let (intermediate_batch, _) = build_batch_from_indices(
+        filter.schema(),
+        left,
+        right,
+        PrimitiveArray::from(left_indices.data().clone()),
+        PrimitiveArray::from(right_indices.data().clone()),
+        filter.column_indices(),
+    )?;
+
+    match join_type {
+        JoinType::Inner | JoinType::Left => {
+            // For both INNER and LEFT joins, input arrays contains only indices for matched data.
+            // Due to this fact it's correct to simply apply filter to intermediate batch and return
+            // indices for left/right rows satisfying filter predicate
+            let filter_result = filter
+                .expression()
+                .evaluate(&intermediate_batch)?
+                .into_array(intermediate_batch.num_rows());
+            let mask = as_boolean_array(&filter_result);
+
+            let left_filtered = PrimitiveArray::<UInt64Type>::from(
+                compute::filter(&left_indices, mask)?.data().clone(),

Review Comment:
   I think here, we apply filter after `build_batch_from_indices` means : apply filter after join operator. Is there a chance filter before join.
   if wrong plz correct me.



##########
datafusion/core/src/physical_plan/join_utils.rs:
##########
@@ -96,6 +97,65 @@ pub struct ColumnIndex {
     pub side: JoinSide,
 }
 
+/// Filter applied before join output
+#[derive(Debug, Clone)]
+pub struct JoinFilter {
+    /// Filter expresssion

Review Comment:
   ```suggestion
       /// Filter expression
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org