You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/02/04 01:35:19 UTC

[PR] refactor join dynamic filters to be more granular [datafusion]

adriangb opened a new pull request, #20142:
URL: https://github.com/apache/datafusion/pull/20142

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#issuecomment-3844808520

   ๐Ÿค– `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing remove-partition-sync (6b948bd7a39f959566862f2c636f1e80a4e6404d) to 81f7a871a99168b51e8317537bc6faa82989bfe9 [diff](https://github.com/apache/datafusion/compare/81f7a871a99168b51e8317537bc6faa82989bfe9..6b948bd7a39f959566862f2c636f1e80a4e6404d) using:  tpch
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#issuecomment-3902280301

   ๐Ÿค–: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   Comparing HEAD and remove-partition-sync
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
   โ”ƒ Query     โ”ƒ        HEAD โ”ƒ remove-partition-sync โ”ƒ        Change โ”ƒ
   โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
   โ”‚ QQuery 1  โ”‚    86.45 ms โ”‚              83.20 ms โ”‚     no change โ”‚
   โ”‚ QQuery 2  โ”‚   241.11 ms โ”‚             251.75 ms โ”‚     no change โ”‚
   โ”‚ QQuery 3  โ”‚   161.11 ms โ”‚             153.38 ms โ”‚     no change โ”‚
   โ”‚ QQuery 4  โ”‚  1666.93 ms โ”‚            1665.95 ms โ”‚     no change โ”‚
   โ”‚ QQuery 5  โ”‚   306.90 ms โ”‚             303.65 ms โ”‚     no change โ”‚
   โ”‚ QQuery 6  โ”‚   427.04 ms โ”‚             430.59 ms โ”‚     no change โ”‚
   โ”‚ QQuery 7  โ”‚   539.19 ms โ”‚             532.81 ms โ”‚     no change โ”‚
   โ”‚ QQuery 8  โ”‚   246.73 ms โ”‚             243.49 ms โ”‚     no change โ”‚
   โ”‚ QQuery 9  โ”‚   870.88 ms โ”‚             833.86 ms โ”‚     no change โ”‚
   โ”‚ QQuery 10 โ”‚   169.94 ms โ”‚             169.37 ms โ”‚     no change โ”‚
   โ”‚ QQuery 11 โ”‚  1044.91 ms โ”‚            1046.28 ms โ”‚     no change โ”‚
   โ”‚ QQuery 12 โ”‚    56.45 ms โ”‚              55.67 ms โ”‚     no change โ”‚
   โ”‚ QQuery 13 โ”‚   860.72 ms โ”‚             868.84 ms โ”‚     no change โ”‚
   โ”‚ QQuery 14 โ”‚  2564.79 ms โ”‚            2530.77 ms โ”‚     no change โ”‚
   โ”‚ QQuery 15 โ”‚    31.99 ms โ”‚              32.25 ms โ”‚     no change โ”‚
   โ”‚ QQuery 16 โ”‚    66.21 ms โ”‚              65.15 ms โ”‚     no change โ”‚
   โ”‚ QQuery 17 โ”‚   301.31 ms โ”‚             295.05 ms โ”‚     no change โ”‚
   โ”‚ QQuery 18 โ”‚   478.23 ms โ”‚             468.35 ms โ”‚     no change โ”‚
   โ”‚ QQuery 19 โ”‚   194.43 ms โ”‚             190.79 ms โ”‚     no change โ”‚
   โ”‚ QQuery 20 โ”‚    27.09 ms โ”‚              26.13 ms โ”‚     no change โ”‚
   โ”‚ QQuery 21 โ”‚    50.55 ms โ”‚              49.45 ms โ”‚     no change โ”‚
   โ”‚ QQuery 22 โ”‚   779.54 ms โ”‚             739.82 ms โ”‚ +1.05x faster โ”‚
   โ”‚ QQuery 23 โ”‚  2498.69 ms โ”‚            2467.70 ms โ”‚     no change โ”‚
   โ”‚ QQuery 24 โ”‚   253.57 ms โ”‚             247.94 ms โ”‚     no change โ”‚
   โ”‚ QQuery 25 โ”‚   451.20 ms โ”‚             445.76 ms โ”‚     no change โ”‚
   โ”‚ QQuery 26 โ”‚   275.58 ms โ”‚             270.75 ms โ”‚     no change โ”‚
   โ”‚ QQuery 27 โ”‚   529.24 ms โ”‚             522.76 ms โ”‚     no change โ”‚
   โ”‚ QQuery 28 โ”‚   503.96 ms โ”‚             515.93 ms โ”‚     no change โ”‚
   โ”‚ QQuery 29 โ”‚   380.31 ms โ”‚             387.92 ms โ”‚     no change โ”‚
   โ”‚ QQuery 30 โ”‚    89.70 ms โ”‚              92.40 ms โ”‚     no change โ”‚
   โ”‚ QQuery 31 โ”‚   284.95 ms โ”‚             284.67 ms โ”‚     no change โ”‚
   โ”‚ QQuery 32 โ”‚    74.90 ms โ”‚              77.18 ms โ”‚     no change โ”‚
   โ”‚ QQuery 33 โ”‚   197.01 ms โ”‚             195.63 ms โ”‚     no change โ”‚
   โ”‚ QQuery 34 โ”‚   197.18 ms โ”‚             195.63 ms โ”‚     no change โ”‚
   โ”‚ QQuery 35 โ”‚   254.87 ms โ”‚             243.43 ms โ”‚     no change โ”‚
   โ”‚ QQuery 36 โ”‚   352.76 ms โ”‚             349.95 ms โ”‚     no change โ”‚
   โ”‚ QQuery 37 โ”‚  1014.61 ms โ”‚             963.60 ms โ”‚ +1.05x faster โ”‚
   โ”‚ QQuery 38 โ”‚   185.19 ms โ”‚             179.37 ms โ”‚     no change โ”‚
   โ”‚ QQuery 39 โ”‚   218.45 ms โ”‚             219.87 ms โ”‚     no change โ”‚
   โ”‚ QQuery 40 โ”‚   191.58 ms โ”‚             193.82 ms โ”‚     no change โ”‚
   โ”‚ QQuery 41 โ”‚    26.77 ms โ”‚              27.73 ms โ”‚     no change โ”‚
   โ”‚ QQuery 42 โ”‚   150.74 ms โ”‚             145.95 ms โ”‚     no change โ”‚
   โ”‚ QQuery 43 โ”‚   166.66 ms โ”‚             163.90 ms โ”‚     no change โ”‚
   โ”‚ QQuery 44 โ”‚    28.19 ms โ”‚              28.71 ms โ”‚     no change โ”‚
   โ”‚ QQuery 45 โ”‚    72.34 ms โ”‚              71.57 ms โ”‚     no change โ”‚
   โ”‚ QQuery 46 โ”‚   322.54 ms โ”‚             322.13 ms โ”‚     no change โ”‚
   โ”‚ QQuery 47 โ”‚  1250.62 ms โ”‚            1267.89 ms โ”‚     no change โ”‚
   โ”‚ QQuery 48 โ”‚   730.87 ms โ”‚             718.73 ms โ”‚     no change โ”‚
   โ”‚ QQuery 49 โ”‚   554.22 ms โ”‚             552.05 ms โ”‚     no change โ”‚
   โ”‚ QQuery 50 โ”‚   742.57 ms โ”‚             729.94 ms โ”‚     no change โ”‚
   โ”‚ QQuery 51 โ”‚   360.80 ms โ”‚             367.99 ms โ”‚     no change โ”‚
   โ”‚ QQuery 52 โ”‚   145.42 ms โ”‚             144.71 ms โ”‚     no change โ”‚
   โ”‚ QQuery 53 โ”‚   210.76 ms โ”‚             208.87 ms โ”‚     no change โ”‚
   โ”‚ QQuery 54 โ”‚   185.56 ms โ”‚             183.98 ms โ”‚     no change โ”‚
   โ”‚ QQuery 55 โ”‚   148.27 ms โ”‚             144.11 ms โ”‚     no change โ”‚
   โ”‚ QQuery 56 โ”‚   195.26 ms โ”‚             190.82 ms โ”‚     no change โ”‚
   โ”‚ QQuery 57 โ”‚   320.54 ms โ”‚             322.13 ms โ”‚     no change โ”‚
   โ”‚ QQuery 58 โ”‚   613.11 ms โ”‚             604.24 ms โ”‚     no change โ”‚
   โ”‚ QQuery 59 โ”‚   368.88 ms โ”‚             370.08 ms โ”‚     no change โ”‚
   โ”‚ QQuery 60 โ”‚   203.57 ms โ”‚             204.37 ms โ”‚     no change โ”‚
   โ”‚ QQuery 61 โ”‚   273.45 ms โ”‚             269.72 ms โ”‚     no change โ”‚
   โ”‚ QQuery 62 โ”‚  1296.36 ms โ”‚            1316.41 ms โ”‚     no change โ”‚
   โ”‚ QQuery 63 โ”‚   213.15 ms โ”‚             215.25 ms โ”‚     no change โ”‚
   โ”‚ QQuery 64 โ”‚ 29569.33 ms โ”‚           29025.33 ms โ”‚     no change โ”‚
   โ”‚ QQuery 65 โ”‚   448.63 ms โ”‚             439.50 ms โ”‚     no change โ”‚
   โ”‚ QQuery 66 โ”‚   287.22 ms โ”‚             288.63 ms โ”‚     no change โ”‚
   โ”‚ QQuery 67 โ”‚   851.07 ms โ”‚             836.32 ms โ”‚     no change โ”‚
   โ”‚ QQuery 68 โ”‚   344.09 ms โ”‚             334.87 ms โ”‚     no change โ”‚
   โ”‚ QQuery 69 โ”‚   168.85 ms โ”‚             172.21 ms โ”‚     no change โ”‚
   โ”‚ QQuery 70 โ”‚   660.77 ms โ”‚             653.29 ms โ”‚     no change โ”‚
   โ”‚ QQuery 71 โ”‚   174.16 ms โ”‚             180.76 ms โ”‚     no change โ”‚
   โ”‚ QQuery 72 โ”‚  1789.62 ms โ”‚            1752.40 ms โ”‚     no change โ”‚
   โ”‚ QQuery 73 โ”‚   174.87 ms โ”‚             187.04 ms โ”‚  1.07x slower โ”‚
   โ”‚ QQuery 74 โ”‚   745.50 ms โ”‚             763.27 ms โ”‚     no change โ”‚
   โ”‚ QQuery 75 โ”‚   456.21 ms โ”‚             454.14 ms โ”‚     no change โ”‚
   โ”‚ QQuery 76 โ”‚   395.82 ms โ”‚             393.62 ms โ”‚     no change โ”‚
   โ”‚ QQuery 77 โ”‚   338.64 ms โ”‚             338.75 ms โ”‚     no change โ”‚
   โ”‚ QQuery 78 โ”‚   686.83 ms โ”‚             696.75 ms โ”‚     no change โ”‚
   โ”‚ QQuery 79 โ”‚   376.68 ms โ”‚             374.47 ms โ”‚     no change โ”‚
   โ”‚ QQuery 80 โ”‚   535.65 ms โ”‚             535.77 ms โ”‚     no change โ”‚
   โ”‚ QQuery 81 โ”‚    59.79 ms โ”‚              59.54 ms โ”‚     no change โ”‚
   โ”‚ QQuery 82 โ”‚   260.98 ms โ”‚             257.87 ms โ”‚     no change โ”‚
   โ”‚ QQuery 83 โ”‚   110.70 ms โ”‚             106.65 ms โ”‚     no change โ”‚
   โ”‚ QQuery 84 โ”‚    96.69 ms โ”‚              96.36 ms โ”‚     no change โ”‚
   โ”‚ QQuery 85 โ”‚   417.88 ms โ”‚             427.38 ms โ”‚     no change โ”‚
   โ”‚ QQuery 86 โ”‚    75.17 ms โ”‚              70.97 ms โ”‚ +1.06x faster โ”‚
   โ”‚ QQuery 87 โ”‚   181.11 ms โ”‚             184.04 ms โ”‚     no change โ”‚
   โ”‚ QQuery 88 โ”‚   313.15 ms โ”‚             308.31 ms โ”‚     no change โ”‚
   โ”‚ QQuery 89 โ”‚   233.12 ms โ”‚             233.84 ms โ”‚     no change โ”‚
   โ”‚ QQuery 90 โ”‚    47.82 ms โ”‚              47.03 ms โ”‚     no change โ”‚
   โ”‚ QQuery 91 โ”‚   147.11 ms โ”‚             152.57 ms โ”‚     no change โ”‚
   โ”‚ QQuery 92 โ”‚    73.23 ms โ”‚              76.29 ms โ”‚     no change โ”‚
   โ”‚ QQuery 93 โ”‚   287.11 ms โ”‚             291.79 ms โ”‚     no change โ”‚
   โ”‚ QQuery 94 โ”‚    95.53 ms โ”‚              95.39 ms โ”‚     no change โ”‚
   โ”‚ QQuery 95 โ”‚   322.24 ms โ”‚             316.76 ms โ”‚     no change โ”‚
   โ”‚ QQuery 96 โ”‚   131.26 ms โ”‚             130.71 ms โ”‚     no change โ”‚
   โ”‚ QQuery 97 โ”‚   239.79 ms โ”‚             232.95 ms โ”‚     no change โ”‚
   โ”‚ QQuery 98 โ”‚   161.73 ms โ”‚             160.75 ms โ”‚     no change โ”‚
   โ”‚ QQuery 99 โ”‚ 14331.37 ms โ”‚           14366.95 ms โ”‚     no change โ”‚
   โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
   โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
   โ”ƒ Benchmark Summary                    โ”ƒ            โ”ƒ
   โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
   โ”‚ Total Time (HEAD)                    โ”‚ 83296.62ms โ”‚
   โ”‚ Total Time (remove-partition-sync)   โ”‚ 82505.45ms โ”‚
   โ”‚ Average Time (HEAD)                  โ”‚   841.38ms โ”‚
   โ”‚ Average Time (remove-partition-sync) โ”‚   833.39ms โ”‚
   โ”‚ Queries Faster                       โ”‚          3 โ”‚
   โ”‚ Queries Slower                       โ”‚          1 โ”‚
   โ”‚ Queries with No Change               โ”‚         95 โ”‚
   โ”‚ Queries with Failure                 โ”‚          0 โ”‚
   โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#discussion_r2770882485


##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -5091,17 +5176,17 @@ mod tests {
             false,
         )?;
         join.dynamic_filter = Some(HashJoinExecDynamicFilter {
-            filter: dynamic_filter,
+            partition_filters,
             build_accumulator: OnceLock::new(),
         });
 
         // Execute the join
         let stream = join.execute(0, task_ctx)?;
         let _batches = common::collect(stream).await?;
 
-        // After the join completes, the dynamic filter should be marked as complete
+        // After the join completes, the partition filter should be marked as complete
         // wait_complete() should return immediately
-        dynamic_filter_clone.wait_complete().await;
+        filter_to_wait.wait_complete().await;

Review Comment:
   > concern is how custom leaf nodes (ExecutionPlan) that have Partitioned HashJoins to fetch the build in parallel but need to know when all partitions have reported their data would adapt to the new structure
   
   This is something you have internally right? Could you help explain how this works / what the requirement is? At some point I wonder if it wouldn't be better to have an optimizer rule that links the join node to the custom scan node if the custom scan node needs to behave in a specific way if it is a the leaf of a join (as opposed to e.g. the current parquet scan node that doesn't care at all)?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#discussion_r2770786736


##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -5091,17 +5176,17 @@ mod tests {
             false,
         )?;
         join.dynamic_filter = Some(HashJoinExecDynamicFilter {
-            filter: dynamic_filter,
+            partition_filters,
             build_accumulator: OnceLock::new(),
         });
 
         // Execute the join
         let stream = join.execute(0, task_ctx)?;
         let _batches = common::collect(stream).await?;
 
-        // After the join completes, the dynamic filter should be marked as complete
+        // After the join completes, the partition filter should be marked as complete
         // wait_complete() should return immediately
-        dynamic_filter_clone.wait_complete().await;
+        filter_to_wait.wait_complete().await;

Review Comment:
   I think this PR makes sense and believe datafusion-distributed could benefit from it a lot. My main concern is how custom leaf nodes (ExecutionPlan) that have Partitioned HashJoins to fetch the build in parallel but need to know when all partitions have reported their data would adapt to the new structure. With this PR, they would need to:
   
   1. Detect the CASE expression
   2. Extract all N DynamicFilters from the branches
   3. Wait for all of them to complete
   
   I wonder if it would be worth keeping a way to know when all partitions have reported their filters. I was wondering if maybe keeping the whole `CASE` expression wrapped in a `DynamicFilterPhysicalExpr` and then having the nested `DynamicFilterPhysicalExpr` would work, although I guess this is not the cleanest approach because how would we differentiate between the outermost `DynamicFilterPhysicalExpr` and a branch `DynamicFilterPhysicalExpr`?  ๐Ÿค” Another option I could think of is: what if we have two types of `DynamicFilterExpr`? I imagine `DynamicFilterPhysicalExpr` for CollectLeft and partitioned branch dynamic filters, and then something like `PartitionedDynamicFilter` that has a `Vec<Arc<DynamicFilterPhysicalExpr>>` (this is just a quick thought)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#discussion_r2767479724


##########
datafusion/core/tests/physical_optimizer/filter_pushdown.rs:
##########
@@ -1247,14 +1247,12 @@ async fn test_hashjoin_dynamic_filter_pushdown_partitioned() {
     -       RepartitionExec: partitioning=Hash([a@0, b@1], 12), input_partitions=1
     -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, c], file_type=test, pushdown_supported=true
     -       RepartitionExec: partitioning=Hash([a@0, b@1], 12), input_partitions=1
-    -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, e], file_type=test, pushdown_supported=true, predicate=DynamicFilter [ CASE hash_repartition % 12 WHEN 2 THEN a@0 >= ab AND a@0 <= ab AND b@1 >= bb AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:ab,c1:bb}]) WHEN 4 THEN a@0 >= aa AND a@0 <= aa AND b@1 >= ba AND b@1 <= ba AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}]) ELSE false END ]
+    -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, e], file_type=test, pushdown_supported=true, predicate=CASE hash_repartition % 12 WHEN 0 THEN DynamicFilter [ false ] WHEN 1 THEN DynamicFilter [ false ] WHEN 2 THEN DynamicFilter [ a@0 >= ab AND a@0 <= ab AND b@1 >= bb AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:ab,c1:bb}]) ] WHEN 3 THEN DynamicFilter [ false ] WHEN 4 THEN DynamicFilter [ a@0 >= aa AND a@0 <= aa AND b@1 >= ba AND b@1 <= ba AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}]) ] WHEN 5 THEN DynamicFilter [ false ] WHEN 6 THEN DynamicFilter [ false ] WHEN 7 THEN DynamicFilter [ false ] WHEN 8 THEN DynamicFilter [ false ] WHEN 9 THEN DynamicFilter [ false ] WHEN 10 THEN DynamicFilter [ false ] WHEN 11 THEN DynamicFilter [ false ] ELSE false END

Review Comment:
   ```b@1 >= bb AND b@1 <= bb```
   It seems it doesn't simplify this yet to `b@1 = bb` (there are quite some examples of this)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#discussion_r2770871468


##########
datafusion/core/tests/physical_optimizer/filter_pushdown.rs:
##########
@@ -1213,7 +1213,7 @@ async fn test_hashjoin_dynamic_filter_pushdown_partitioned() {
           -       RepartitionExec: partitioning=Hash([a@0, b@1], 12), input_partitions=1
           -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, c], file_type=test, pushdown_supported=true
           -       RepartitionExec: partitioning=Hash([a@0, b@1], 12), input_partitions=1
-          -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, e], file_type=test, pushdown_supported=true, predicate=DynamicFilter [ empty ]
+          -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, e], file_type=test, pushdown_supported=true, predicate=CASE hash_repartition % 12 WHEN 0 THEN DynamicFilter [ empty ] WHEN 1 THEN DynamicFilter [ empty ] WHEN 2 THEN DynamicFilter [ empty ] WHEN 3 THEN DynamicFilter [ empty ] WHEN 4 THEN DynamicFilter [ empty ] WHEN 5 THEN DynamicFilter [ empty ] WHEN 6 THEN DynamicFilter [ empty ] WHEN 7 THEN DynamicFilter [ empty ] WHEN 8 THEN DynamicFilter [ empty ] WHEN 9 THEN DynamicFilter [ empty ] WHEN 10 THEN DynamicFilter [ empty ] WHEN 11 THEN DynamicFilter [ empty ] ELSE false END

Review Comment:
   One thing I want to be contentious of is to try to keep the expressions such that it can be used for stats pruning



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#discussion_r2767498605


##########
datafusion/core/tests/physical_optimizer/filter_pushdown.rs:
##########
@@ -1247,14 +1247,12 @@ async fn test_hashjoin_dynamic_filter_pushdown_partitioned() {
     -       RepartitionExec: partitioning=Hash([a@0, b@1], 12), input_partitions=1
     -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, c], file_type=test, pushdown_supported=true
     -       RepartitionExec: partitioning=Hash([a@0, b@1], 12), input_partitions=1
-    -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, e], file_type=test, pushdown_supported=true, predicate=DynamicFilter [ CASE hash_repartition % 12 WHEN 2 THEN a@0 >= ab AND a@0 <= ab AND b@1 >= bb AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:ab,c1:bb}]) WHEN 4 THEN a@0 >= aa AND a@0 <= aa AND b@1 >= ba AND b@1 <= ba AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}]) ELSE false END ]
+    -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, e], file_type=test, pushdown_supported=true, predicate=CASE hash_repartition % 12 WHEN 0 THEN DynamicFilter [ false ] WHEN 1 THEN DynamicFilter [ false ] WHEN 2 THEN DynamicFilter [ a@0 >= ab AND a@0 <= ab AND b@1 >= bb AND b@1 <= bb AND struct(a@0, b@1) IN (SET) ([{c0:ab,c1:bb}]) ] WHEN 3 THEN DynamicFilter [ false ] WHEN 4 THEN DynamicFilter [ a@0 >= aa AND a@0 <= aa AND b@1 >= ba AND b@1 <= ba AND struct(a@0, b@1) IN (SET) ([{c0:aa,c1:ba}]) ] WHEN 5 THEN DynamicFilter [ false ] WHEN 6 THEN DynamicFilter [ false ] WHEN 7 THEN DynamicFilter [ false ] WHEN 8 THEN DynamicFilter [ false ] WHEN 9 THEN DynamicFilter [ false ] WHEN 10 THEN DynamicFilter [ false ] WHEN 11 THEN DynamicFilter [ false ] ELSE false END

Review Comment:
   Also the the `IN (SET)` could be removed in this case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#discussion_r2771436549


##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -5091,17 +5176,17 @@ mod tests {
             false,
         )?;
         join.dynamic_filter = Some(HashJoinExecDynamicFilter {
-            filter: dynamic_filter,
+            partition_filters,
             build_accumulator: OnceLock::new(),
         });
 
         // Execute the join
         let stream = join.execute(0, task_ctx)?;
         let _batches = common::collect(stream).await?;
 
-        // After the join completes, the dynamic filter should be marked as complete
+        // After the join completes, the partition filter should be marked as complete
         // wait_complete() should return immediately
-        dynamic_filter_clone.wait_complete().await;
+        filter_to_wait.wait_complete().await;

Review Comment:
   Yeah, we have some custom `ExecutionPlan` leaf nodes that don't evaluate filters in the leaf node itself but instead push query execution (along with the filters) to a remote data source. The key requirement is that we need to wait for all build partitions to report their filters before querying the remote source because we execute this leaf node in a single partition (even if we have a Partitioned join), so we need the complete filter information from all hash join build partitions to generate a correct predicate.
   
   I was raising it in case other custom scans had similar patterns, but if it's too specific to our use case, we can handle it internally.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#discussion_r2761815407


##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -479,12 +482,18 @@ pub struct HashJoinExec {
     /// Set when dynamic filter pushdown is detected in handle_child_pushdown_result.
     /// HashJoinExec also needs to keep a shared bounds accumulator for coordinating updates.
     dynamic_filter: Option<HashJoinExecDynamicFilter>,
+    /// Temporary storage for per-partition DynamicFilters during filter pushdown.
+    /// Set in gather_filters_for_pushdown, consumed in handle_child_pushdown_result.
+    /// Uses Mutex because gather_filters_for_pushdown takes &self and may be called
+    /// multiple times on the same node (e.g., when optimizations are re-run).
+    pending_partition_filters: Mutex<Option<Vec<Arc<DynamicFilterPhysicalExpr>>>>,

Review Comment:
   The alternative to this is to use the values passed into `handle_child_pushdown_result` which *do* include the filters we are looking for but it involves downcast matching through the CASE expression and then cloning DynamicFilterPhysicalExpr



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#issuecomment-3844850555

   ๐Ÿค–: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   Comparing HEAD and remove-partition-sync
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
   โ”ƒ Query     โ”ƒ      HEAD โ”ƒ remove-partition-sync โ”ƒ    Change โ”ƒ
   โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
   โ”‚ QQuery 1  โ”‚ 178.08 ms โ”‚             178.92 ms โ”‚ no change โ”‚
   โ”‚ QQuery 2  โ”‚  88.20 ms โ”‚              86.18 ms โ”‚ no change โ”‚
   โ”‚ QQuery 3  โ”‚ 119.26 ms โ”‚             124.26 ms โ”‚ no change โ”‚
   โ”‚ QQuery 4  โ”‚  78.36 ms โ”‚              78.74 ms โ”‚ no change โ”‚
   โ”‚ QQuery 5  โ”‚ 172.62 ms โ”‚             174.68 ms โ”‚ no change โ”‚
   โ”‚ QQuery 6  โ”‚  69.46 ms โ”‚              68.71 ms โ”‚ no change โ”‚
   โ”‚ QQuery 7  โ”‚ 206.80 ms โ”‚             212.64 ms โ”‚ no change โ”‚
   โ”‚ QQuery 8  โ”‚ 167.54 ms โ”‚             165.31 ms โ”‚ no change โ”‚
   โ”‚ QQuery 9  โ”‚ 225.20 ms โ”‚             228.05 ms โ”‚ no change โ”‚
   โ”‚ QQuery 10 โ”‚ 186.03 ms โ”‚             185.50 ms โ”‚ no change โ”‚
   โ”‚ QQuery 11 โ”‚  62.03 ms โ”‚              60.59 ms โ”‚ no change โ”‚
   โ”‚ QQuery 12 โ”‚ 118.19 ms โ”‚             118.45 ms โ”‚ no change โ”‚
   โ”‚ QQuery 13 โ”‚ 218.12 ms โ”‚             222.84 ms โ”‚ no change โ”‚
   โ”‚ QQuery 14 โ”‚  86.18 ms โ”‚              85.79 ms โ”‚ no change โ”‚
   โ”‚ QQuery 15 โ”‚ 127.24 ms โ”‚             123.31 ms โ”‚ no change โ”‚
   โ”‚ QQuery 16 โ”‚  60.54 ms โ”‚              59.65 ms โ”‚ no change โ”‚
   โ”‚ QQuery 17 โ”‚ 255.75 ms โ”‚             263.53 ms โ”‚ no change โ”‚
   โ”‚ QQuery 18 โ”‚ 306.91 ms โ”‚             309.43 ms โ”‚ no change โ”‚
   โ”‚ QQuery 19 โ”‚ 131.24 ms โ”‚             134.50 ms โ”‚ no change โ”‚
   โ”‚ QQuery 20 โ”‚ 127.85 ms โ”‚             128.78 ms โ”‚ no change โ”‚
   โ”‚ QQuery 21 โ”‚ 259.38 ms โ”‚             260.41 ms โ”‚ no change โ”‚
   โ”‚ QQuery 22 โ”‚  41.81 ms โ”‚              41.02 ms โ”‚ no change โ”‚
   โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
   โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
   โ”ƒ Benchmark Summary                    โ”ƒ           โ”ƒ
   โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
   โ”‚ Total Time (HEAD)                    โ”‚ 3286.79ms โ”‚
   โ”‚ Total Time (remove-partition-sync)   โ”‚ 3311.28ms โ”‚
   โ”‚ Average Time (HEAD)                  โ”‚  149.40ms โ”‚
   โ”‚ Average Time (remove-partition-sync) โ”‚  150.51ms โ”‚
   โ”‚ Queries Faster                       โ”‚         0 โ”‚
   โ”‚ Queries Slower                       โ”‚         0 โ”‚
   โ”‚ Queries with No Change               โ”‚        22 โ”‚
   โ”‚ Queries with Failure                 โ”‚         0 โ”‚
   โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#issuecomment-3902223187

   ๐Ÿค– `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing remove-partition-sync (39e613343664441454a0c77b08e599402eb09a39) to 7d217b1ec3bd4d447961e84bc3fe7f7708546b2d [diff](https://github.com/apache/datafusion/compare/7d217b1ec3bd4d447961e84bc3fe7f7708546b2d..39e613343664441454a0c77b08e599402eb09a39) using:  tpcds
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#issuecomment-3844867689

   @LiaCastaneda @Dandandan wonder what you folks think of this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#discussion_r2767493460


##########
datafusion/core/tests/physical_optimizer/filter_pushdown.rs:
##########
@@ -1213,7 +1213,7 @@ async fn test_hashjoin_dynamic_filter_pushdown_partitioned() {
           -       RepartitionExec: partitioning=Hash([a@0, b@1], 12), input_partitions=1
           -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, c], file_type=test, pushdown_supported=true
           -       RepartitionExec: partitioning=Hash([a@0, b@1], 12), input_partitions=1
-          -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, e], file_type=test, pushdown_supported=true, predicate=DynamicFilter [ empty ]
+          -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, e], file_type=test, pushdown_supported=true, predicate=CASE hash_repartition % 12 WHEN 0 THEN DynamicFilter [ empty ] WHEN 1 THEN DynamicFilter [ empty ] WHEN 2 THEN DynamicFilter [ empty ] WHEN 3 THEN DynamicFilter [ empty ] WHEN 4 THEN DynamicFilter [ empty ] WHEN 5 THEN DynamicFilter [ empty ] WHEN 6 THEN DynamicFilter [ empty ] WHEN 7 THEN DynamicFilter [ empty ] WHEN 8 THEN DynamicFilter [ empty ] WHEN 9 THEN DynamicFilter [ empty ] WHEN 10 THEN DynamicFilter [ empty ] WHEN 11 THEN DynamicFilter [ empty ] ELSE false END

Review Comment:
   OTOH I guess this expression can simplify those inner expression itself.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#discussion_r2771436549


##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -5091,17 +5176,17 @@ mod tests {
             false,
         )?;
         join.dynamic_filter = Some(HashJoinExecDynamicFilter {
-            filter: dynamic_filter,
+            partition_filters,
             build_accumulator: OnceLock::new(),
         });
 
         // Execute the join
         let stream = join.execute(0, task_ctx)?;
         let _batches = common::collect(stream).await?;
 
-        // After the join completes, the dynamic filter should be marked as complete
+        // After the join completes, the partition filter should be marked as complete
         // wait_complete() should return immediately
-        dynamic_filter_clone.wait_complete().await;
+        filter_to_wait.wait_complete().await;

Review Comment:
   Yeah, we have some custom `ExecutionPlan` leaf nodes that don't evaluate filters in the leaf node itself but instead push query execution (along with the filters) to a remote data source. The key requirement is that we need to wait for all build partitions to report their filters before querying the remote source because we execute this leaf node in a single partition (even if we have a Partitioned join), so we need the complete filter information from all hash join build partitions to generate a correct predicate.
   
   I was raising it in case other custom scans had similar patterns, but if it's too specific to our use case, we can handle it internally.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#issuecomment-3902167953

   run benchmark tpcds
   DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS=true
   DATAFUSION_EXECUTION_PARQUET_REORDER_FILTERS=true


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#issuecomment-3844850658

   ๐Ÿค– `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing remove-partition-sync (6b948bd7a39f959566862f2c636f1e80a4e6404d) to 81f7a871a99168b51e8317537bc6faa82989bfe9 [diff](https://github.com/apache/datafusion/compare/81f7a871a99168b51e8317537bc6faa82989bfe9..6b948bd7a39f959566862f2c636f1e80a4e6404d) using:  tpcds
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#discussion_r2767474039


##########
datafusion/core/tests/physical_optimizer/filter_pushdown.rs:
##########
@@ -1213,7 +1213,7 @@ async fn test_hashjoin_dynamic_filter_pushdown_partitioned() {
           -       RepartitionExec: partitioning=Hash([a@0, b@1], 12), input_partitions=1
           -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, c], file_type=test, pushdown_supported=true
           -       RepartitionExec: partitioning=Hash([a@0, b@1], 12), input_partitions=1
-          -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, e], file_type=test, pushdown_supported=true, predicate=DynamicFilter [ empty ]
+          -         DataSourceExec: file_groups={1 group: [[test.parquet]]}, projection=[a, b, e], file_type=test, pushdown_supported=true, predicate=CASE hash_repartition % 12 WHEN 0 THEN DynamicFilter [ empty ] WHEN 1 THEN DynamicFilter [ empty ] WHEN 2 THEN DynamicFilter [ empty ] WHEN 3 THEN DynamicFilter [ empty ] WHEN 4 THEN DynamicFilter [ empty ] WHEN 5 THEN DynamicFilter [ empty ] WHEN 6 THEN DynamicFilter [ empty ] WHEN 7 THEN DynamicFilter [ empty ] WHEN 8 THEN DynamicFilter [ empty ] WHEN 9 THEN DynamicFilter [ empty ] WHEN 10 THEN DynamicFilter [ empty ] WHEN 11 THEN DynamicFilter [ empty ] ELSE false END

Review Comment:
   I wonder if the routing can be better done by some custom `PhysicalExpr` rather than the long `CASE WHEN`. It at least allows for some easier optimization (e.g. doing direct lookup instead of expression evaluation), the downside is maybe that expression simplification won't work as automatic?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#discussion_r2770786736


##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -5091,17 +5176,17 @@ mod tests {
             false,
         )?;
         join.dynamic_filter = Some(HashJoinExecDynamicFilter {
-            filter: dynamic_filter,
+            partition_filters,
             build_accumulator: OnceLock::new(),
         });
 
         // Execute the join
         let stream = join.execute(0, task_ctx)?;
         let _batches = common::collect(stream).await?;
 
-        // After the join completes, the dynamic filter should be marked as complete
+        // After the join completes, the partition filter should be marked as complete
         // wait_complete() should return immediately
-        dynamic_filter_clone.wait_complete().await;
+        filter_to_wait.wait_complete().await;

Review Comment:
   I think this PR makes sense and believe datafusion-distributed could benefit from it a lot. My main concern is how custom leaf nodes (ExecutionPlan) that have Partitioned HashJoins to fetch the build in parallel but need to know when all partitions have reported their data would adapt to the new structure. With this PR, they would need to:
   
   1. Detect the CASE expression
   2. Extract all N DynamicFilters from the branches
   3. Wait for all of them to complete
   
   I wonder if it would be worth keeping a way to know when all partitions have reported their filters. I was wondering if maybe keeping the whole `CASE` expression wrapped in a `DynamicFilterPhysicalExpr` and then having the nested `DynamicFilterPhysicalExpr`s per branch would work, although I guess this is not the cleanest approach because how would we differentiate between the outermost `DynamicFilterPhysicalExpr` and a branch `DynamicFilterPhysicalExpr`?  ๐Ÿค” Another option I could think of is: what if we have two types of `DynamicFilterExpr`? I imagine `DynamicFilterPhysicalExpr` for CollectLeft and partitioned branch dynamic filters, and then something like `PartitionedDynamicFilter` that has a `Vec<Arc<DynamicFilterPhysicalExpr>>` (this is just a quick thought)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#issuecomment-3844873829

   ๐Ÿค–: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   Comparing HEAD and remove-partition-sync
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
   โ”ƒ Query     โ”ƒ        HEAD โ”ƒ remove-partition-sync โ”ƒ        Change โ”ƒ
   โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
   โ”‚ QQuery 1  โ”‚    75.66 ms โ”‚              73.70 ms โ”‚     no change โ”‚
   โ”‚ QQuery 2  โ”‚   218.39 ms โ”‚             211.45 ms โ”‚     no change โ”‚
   โ”‚ QQuery 3  โ”‚   157.48 ms โ”‚             156.32 ms โ”‚     no change โ”‚
   โ”‚ QQuery 4  โ”‚  1976.51 ms โ”‚            1926.46 ms โ”‚     no change โ”‚
   โ”‚ QQuery 5  โ”‚   265.29 ms โ”‚             263.97 ms โ”‚     no change โ”‚
   โ”‚ QQuery 6  โ”‚  1468.72 ms โ”‚            1465.84 ms โ”‚     no change โ”‚
   โ”‚ QQuery 7  โ”‚   524.92 ms โ”‚             523.79 ms โ”‚     no change โ”‚
   โ”‚ QQuery 8  โ”‚   170.01 ms โ”‚             171.71 ms โ”‚     no change โ”‚
   โ”‚ QQuery 9  โ”‚   294.66 ms โ”‚             314.46 ms โ”‚  1.07x slower โ”‚
   โ”‚ QQuery 10 โ”‚   174.26 ms โ”‚             174.27 ms โ”‚     no change โ”‚
   โ”‚ QQuery 11 โ”‚  1330.37 ms โ”‚            1322.60 ms โ”‚     no change โ”‚
   โ”‚ QQuery 12 โ”‚    70.17 ms โ”‚              68.56 ms โ”‚     no change โ”‚
   โ”‚ QQuery 13 โ”‚   547.87 ms โ”‚             546.08 ms โ”‚     no change โ”‚
   โ”‚ QQuery 14 โ”‚  1896.83 ms โ”‚            1861.62 ms โ”‚     no change โ”‚
   โ”‚ QQuery 15 โ”‚    28.00 ms โ”‚              26.49 ms โ”‚ +1.06x faster โ”‚
   โ”‚ QQuery 16 โ”‚    65.55 ms โ”‚              64.99 ms โ”‚     no change โ”‚
   โ”‚ QQuery 17 โ”‚   366.68 ms โ”‚             359.47 ms โ”‚     no change โ”‚
   โ”‚ QQuery 18 โ”‚   195.75 ms โ”‚             190.81 ms โ”‚     no change โ”‚
   โ”‚ QQuery 19 โ”‚   219.40 ms โ”‚             215.71 ms โ”‚     no change โ”‚
   โ”‚ QQuery 20 โ”‚    26.18 ms โ”‚              24.30 ms โ”‚ +1.08x faster โ”‚
   โ”‚ QQuery 21 โ”‚    37.03 ms โ”‚              36.42 ms โ”‚     no change โ”‚
   โ”‚ QQuery 22 โ”‚   701.56 ms โ”‚             741.81 ms โ”‚  1.06x slower โ”‚
   โ”‚ QQuery 23 โ”‚  1793.82 ms โ”‚            1779.24 ms โ”‚     no change โ”‚
   โ”‚ QQuery 24 โ”‚   692.37 ms โ”‚             681.06 ms โ”‚     no change โ”‚
   โ”‚ QQuery 25 โ”‚   533.75 ms โ”‚             516.63 ms โ”‚     no change โ”‚
   โ”‚ QQuery 26 โ”‚   129.52 ms โ”‚             126.94 ms โ”‚     no change โ”‚
   โ”‚ QQuery 27 โ”‚   518.90 ms โ”‚             514.48 ms โ”‚     no change โ”‚
   โ”‚ QQuery 28 โ”‚   299.26 ms โ”‚             299.93 ms โ”‚     no change โ”‚
   โ”‚ QQuery 29 โ”‚   451.06 ms โ”‚             452.60 ms โ”‚     no change โ”‚
   โ”‚ QQuery 30 โ”‚    77.06 ms โ”‚              75.65 ms โ”‚     no change โ”‚
   โ”‚ QQuery 31 โ”‚   311.32 ms โ”‚             304.89 ms โ”‚     no change โ”‚
   โ”‚ QQuery 32 โ”‚    81.64 ms โ”‚              81.45 ms โ”‚     no change โ”‚
   โ”‚ QQuery 33 โ”‚   208.35 ms โ”‚             205.51 ms โ”‚     no change โ”‚
   โ”‚ QQuery 34 โ”‚   161.07 ms โ”‚             160.04 ms โ”‚     no change โ”‚
   โ”‚ QQuery 35 โ”‚   183.57 ms โ”‚             175.26 ms โ”‚     no change โ”‚
   โ”‚ QQuery 36 โ”‚   288.23 ms โ”‚             287.71 ms โ”‚     no change โ”‚
   โ”‚ QQuery 37 โ”‚   260.83 ms โ”‚             255.52 ms โ”‚     no change โ”‚
   โ”‚ QQuery 38 โ”‚   161.54 ms โ”‚             153.56 ms โ”‚     no change โ”‚
   โ”‚ QQuery 39 โ”‚   205.13 ms โ”‚             196.78 ms โ”‚     no change โ”‚
   โ”‚ QQuery 40 โ”‚   180.23 ms โ”‚             184.17 ms โ”‚     no change โ”‚
   โ”‚ QQuery 41 โ”‚    25.71 ms โ”‚              25.88 ms โ”‚     no change โ”‚
   โ”‚ QQuery 42 โ”‚   145.72 ms โ”‚             143.16 ms โ”‚     no change โ”‚
   โ”‚ QQuery 43 โ”‚   127.71 ms โ”‚             127.54 ms โ”‚     no change โ”‚
   โ”‚ QQuery 44 โ”‚    29.17 ms โ”‚              28.84 ms โ”‚     no change โ”‚
   โ”‚ QQuery 45 โ”‚    85.39 ms โ”‚              84.84 ms โ”‚     no change โ”‚
   โ”‚ QQuery 46 โ”‚   323.66 ms โ”‚             323.39 ms โ”‚     no change โ”‚
   โ”‚ QQuery 47 โ”‚  1030.34 ms โ”‚            1041.27 ms โ”‚     no change โ”‚
   โ”‚ QQuery 48 โ”‚   406.14 ms โ”‚             408.25 ms โ”‚     no change โ”‚
   โ”‚ QQuery 49 โ”‚   383.50 ms โ”‚             381.52 ms โ”‚     no change โ”‚
   โ”‚ QQuery 50 โ”‚   330.10 ms โ”‚             326.48 ms โ”‚     no change โ”‚
   โ”‚ QQuery 51 โ”‚   309.26 ms โ”‚             303.82 ms โ”‚     no change โ”‚
   โ”‚ QQuery 52 โ”‚   146.79 ms โ”‚             145.49 ms โ”‚     no change โ”‚
   โ”‚ QQuery 53 โ”‚   144.94 ms โ”‚             147.01 ms โ”‚     no change โ”‚
   โ”‚ QQuery 54 โ”‚   204.94 ms โ”‚             202.63 ms โ”‚     no change โ”‚
   โ”‚ QQuery 55 โ”‚   145.51 ms โ”‚             144.83 ms โ”‚     no change โ”‚
   โ”‚ QQuery 56 โ”‚   206.54 ms โ”‚             205.21 ms โ”‚     no change โ”‚
   โ”‚ QQuery 57 โ”‚   290.83 ms โ”‚             290.71 ms โ”‚     no change โ”‚
   โ”‚ QQuery 58 โ”‚   496.29 ms โ”‚             490.44 ms โ”‚     no change โ”‚
   โ”‚ QQuery 59 โ”‚   299.67 ms โ”‚             285.23 ms โ”‚     no change โ”‚
   โ”‚ QQuery 60 โ”‚   213.55 ms โ”‚             210.25 ms โ”‚     no change โ”‚
   โ”‚ QQuery 61 โ”‚   248.68 ms โ”‚             245.38 ms โ”‚     no change โ”‚
   โ”‚ QQuery 62 โ”‚  1272.64 ms โ”‚            1301.13 ms โ”‚     no change โ”‚
   โ”‚ QQuery 63 โ”‚   147.17 ms โ”‚             146.80 ms โ”‚     no change โ”‚
   โ”‚ QQuery 64 โ”‚  1154.36 ms โ”‚            1174.16 ms โ”‚     no change โ”‚
   โ”‚ QQuery 65 โ”‚   356.67 ms โ”‚             352.59 ms โ”‚     no change โ”‚
   โ”‚ QQuery 66 โ”‚   396.88 ms โ”‚             394.95 ms โ”‚     no change โ”‚
   โ”‚ QQuery 67 โ”‚   529.54 ms โ”‚             549.35 ms โ”‚     no change โ”‚
   โ”‚ QQuery 68 โ”‚   375.14 ms โ”‚             374.55 ms โ”‚     no change โ”‚
   โ”‚ QQuery 69 โ”‚   172.39 ms โ”‚             169.51 ms โ”‚     no change โ”‚
   โ”‚ QQuery 70 โ”‚   495.28 ms โ”‚             495.15 ms โ”‚     no change โ”‚
   โ”‚ QQuery 71 โ”‚   186.94 ms โ”‚             183.39 ms โ”‚     no change โ”‚
   โ”‚ QQuery 72 โ”‚  2103.30 ms โ”‚            2114.78 ms โ”‚     no change โ”‚
   โ”‚ QQuery 73 โ”‚   154.89 ms โ”‚             153.48 ms โ”‚     no change โ”‚
   โ”‚ QQuery 74 โ”‚   823.54 ms โ”‚             818.41 ms โ”‚     no change โ”‚
   โ”‚ QQuery 75 โ”‚   417.42 ms โ”‚             414.24 ms โ”‚     no change โ”‚
   โ”‚ QQuery 76 โ”‚   186.48 ms โ”‚             184.51 ms โ”‚     no change โ”‚
   โ”‚ QQuery 77 โ”‚   281.42 ms โ”‚             278.82 ms โ”‚     no change โ”‚
   โ”‚ QQuery 78 โ”‚   683.21 ms โ”‚             680.35 ms โ”‚     no change โ”‚
   โ”‚ QQuery 79 โ”‚   329.96 ms โ”‚             325.68 ms โ”‚     no change โ”‚
   โ”‚ QQuery 80 โ”‚   522.42 ms โ”‚             531.32 ms โ”‚     no change โ”‚
   โ”‚ QQuery 81 โ”‚    51.21 ms โ”‚              54.50 ms โ”‚  1.06x slower โ”‚
   โ”‚ QQuery 82 โ”‚   278.61 ms โ”‚             281.34 ms โ”‚     no change โ”‚
   โ”‚ QQuery 83 โ”‚    79.24 ms โ”‚              79.36 ms โ”‚     no change โ”‚
   โ”‚ QQuery 84 โ”‚    71.85 ms โ”‚              69.03 ms โ”‚     no change โ”‚
   โ”‚ QQuery 85 โ”‚   225.92 ms โ”‚             224.50 ms โ”‚     no change โ”‚
   โ”‚ QQuery 86 โ”‚    59.54 ms โ”‚              58.38 ms โ”‚     no change โ”‚
   โ”‚ QQuery 87 โ”‚   151.96 ms โ”‚             157.63 ms โ”‚     no change โ”‚
   โ”‚ QQuery 88 โ”‚   259.24 ms โ”‚             256.79 ms โ”‚     no change โ”‚
   โ”‚ QQuery 89 โ”‚   163.22 ms โ”‚             168.53 ms โ”‚     no change โ”‚
   โ”‚ QQuery 90 โ”‚    45.15 ms โ”‚              45.03 ms โ”‚     no change โ”‚
   โ”‚ QQuery 91 โ”‚    98.50 ms โ”‚              98.01 ms โ”‚     no change โ”‚
   โ”‚ QQuery 92 โ”‚    82.04 ms โ”‚              80.61 ms โ”‚     no change โ”‚
   โ”‚ QQuery 93 โ”‚   280.43 ms โ”‚             287.26 ms โ”‚     no change โ”‚
   โ”‚ QQuery 94 โ”‚    90.75 ms โ”‚              88.58 ms โ”‚     no change โ”‚
   โ”‚ QQuery 95 โ”‚   243.69 ms โ”‚             246.48 ms โ”‚     no change โ”‚
   โ”‚ QQuery 96 โ”‚   112.03 ms โ”‚             115.40 ms โ”‚     no change โ”‚
   โ”‚ QQuery 97 โ”‚   193.46 ms โ”‚             192.93 ms โ”‚     no change โ”‚
   โ”‚ QQuery 98 โ”‚   221.58 ms โ”‚             219.11 ms โ”‚     no change โ”‚
   โ”‚ QQuery 99 โ”‚ 14202.63 ms โ”‚           14213.60 ms โ”‚     no change โ”‚
   โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
   โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“
   โ”ƒ Benchmark Summary                    โ”ƒ            โ”ƒ
   โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ
   โ”‚ Total Time (HEAD)                    โ”‚ 50444.06ms โ”‚
   โ”‚ Total Time (remove-partition-sync)   โ”‚ 50334.64ms โ”‚
   โ”‚ Average Time (HEAD)                  โ”‚   509.54ms โ”‚
   โ”‚ Average Time (remove-partition-sync) โ”‚   508.43ms โ”‚
   โ”‚ Queries Faster                       โ”‚          2 โ”‚
   โ”‚ Queries Slower                       โ”‚          3 โ”‚
   โ”‚ Queries with No Change               โ”‚         94 โ”‚
   โ”‚ Queries with Failure                 โ”‚          0 โ”‚
   โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#issuecomment-4317539432

   Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #20142: refactor join dynamic filters to be more granular
URL: https://github.com/apache/datafusion/pull/20142


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#discussion_r2761786757


##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -1349,15 +1432,11 @@ impl ExecutionPlan for HashJoinExec {
         let mut result = FilterPushdownPropagation::if_any(child_pushdown_result.clone());
         assert_eq!(child_pushdown_result.self_filters.len(), 2); // Should always be 2, we have 2 children
         let right_child_self_filters = &child_pushdown_result.self_filters[1]; // We only push down filters to the right child
-        // We expect 0 or 1 self filters
-        if let Some(filter) = right_child_self_filters.first() {
-            // Note that we don't check PushdDownPredicate::discrimnant because even if nothing said
-            // "yes, I can fully evaluate this filter" things might still use it for statistics -> it's worth updating
-            let predicate = Arc::clone(&filter.predicate);
-            if let Ok(dynamic_filter) =
-                Arc::downcast::<DynamicFilterPhysicalExpr>(predicate)
-            {
-                // We successfully pushed down our self filter - we need to make a new node with the dynamic filter
+
+        // Check if our self-filter was pushed (pending_partition_filters was set in gather_filters_for_pushdown)
+        if let Some(partition_filters) = self.pending_partition_filters.lock().take() {
+            if right_child_self_filters.first().is_some() {
+                // Self-filter was pushed โ€” create a new node with the partition filters
                 let new_node = Arc::new(HashJoinExec {

Review Comment:
   `right_child_self_filters.first().is_some()` doesn't indicate the self-filter was accepted by the right child; it will be `Some` whenever HashJoin added a self-filter, even if the child marked it unsupported. This can cause the join to treat dynamic filter pushdown as successful and keep `partition_filters` even when they are not referenced by the probe side. Use the `PushedDownPredicate.discriminant` (e.g. require `PushedDown::Yes`) and/or match the specific predicate you pushed before creating `HashJoinExecDynamicFilter`.



##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -1078,20 +1162,13 @@ impl ExecutionPlan for HashJoinExec {
 
         // Only enable dynamic filter pushdown if:
         // - The session config enables dynamic filter pushdown
-        // - A dynamic filter exists
-        // - At least one consumer is holding a reference to it, this avoids expensive filter
-        //   computation when disabled or when no consumer will use it.
+        // - A dynamic filter exists (it was pushed down successfully)
         let enable_dynamic_filter_pushdown = context
             .session_config()
             .options()
             .optimizer
             .enable_join_dynamic_filter_pushdown
-            && self
-                .dynamic_filter
-                .as_ref()
-                .map(|df| df.filter.is_used())
-                .unwrap_or(false);
-
+            && self.dynamic_filter.is_some();

Review Comment:
   `enable_dynamic_filter_pushdown` now checks only `self.dynamic_filter.is_some()`. However `dynamic_filter` can be present even when the probe side doesn't actually hold a reference to the filters (e.g. pushdown unsupported), in which case the build-side will still compute/update dynamic filters unnecessarily. Consider restoring the previous gating by checking whether any `partition_filters` are actually used (e.g. `df.partition_filters.iter().any(|f| f.is_used())`), or alternatively only setting `dynamic_filter` when the self-filter pushdown is reported as supported.



##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -598,15 +607,83 @@ impl HashJoinExec {
             null_aware,
             cache,
             dynamic_filter: None,
+            pending_partition_filters: Mutex::new(None),
         })
     }
 
-    fn create_dynamic_filter(on: &JoinOn) -> Arc<DynamicFilterPhysicalExpr> {
-        // Extract the right-side keys (probe side keys) from the `on` clauses
-        // Dynamic filter will be created from build side values (left side) and applied to probe side (right side)
+    /// Creates per-partition DynamicFilters and the expression to push down to the scan.
+    ///
+    /// Returns `(expression_to_push_down, partition_filters)`:
+    /// - CollectLeft or single partition: returns a single DynamicFilter as both
+    /// - Partitioned N>1: creates N DynamicFilters, builds a CASE expression,
+    ///   returns (case_expr, vec_of_filters)
+    fn create_partition_filters(
+        on: &JoinOn,
+        mode: PartitionMode,
+        num_build_partitions: usize,
+        repartition_random_state: SeededRandomState,
+    ) -> (Arc<dyn PhysicalExpr>, Vec<Arc<DynamicFilterPhysicalExpr>>) {
         let right_keys: Vec<_> = on.iter().map(|(_, r)| Arc::clone(r)).collect();
-        // Initialize with a placeholder expression (true) that will be updated when the hash table is built
-        Arc::new(DynamicFilterPhysicalExpr::new(right_keys, lit(true)))
+
+        if mode == PartitionMode::CollectLeft || num_build_partitions == 1 {
+            // Single filter: used directly as the pushed-down expression
+            let filter = Arc::new(DynamicFilterPhysicalExpr::new(right_keys, lit(true)));
+            let push_expr = Arc::clone(&filter) as Arc<dyn PhysicalExpr>;
+            (push_expr, vec![filter])
+        } else {
+            // Create N per-partition DynamicFilterPhysicalExpr instances,
+            // each starting with lit(true) as a pass-through.
+            let partition_filters: Vec<Arc<DynamicFilterPhysicalExpr>> = (0
+                ..num_build_partitions)
+                .map(|_| {
+                    Arc::new(DynamicFilterPhysicalExpr::new(
+                        right_keys.clone(),
+                        lit(true),
+                    ))
+                })
+                .collect();
+
+            // Build the static CASE expression:
+            // CASE (hash_repartition(join_keys) % N)
+            //   WHEN 0 THEN partition_filters[0]
+            //   WHEN 1 THEN partition_filters[1]
+            //   ...
+            //   ELSE false
+            // END
+            let routing_hash_expr = Arc::new(HashExpr::new(
+                right_keys,
+                repartition_random_state,
+                "hash_repartition".to_string(),
+            )) as Arc<dyn PhysicalExpr>;
+
+            let modulo_expr = Arc::new(BinaryExpr::new(
+                routing_hash_expr,
+                Operator::Modulo,
+                lit(ScalarValue::UInt64(Some(num_build_partitions as u64))),
+            )) as Arc<dyn PhysicalExpr>;
+
+            let when_then_branches: Vec<(Arc<dyn PhysicalExpr>, Arc<dyn PhysicalExpr>)> =
+                partition_filters
+                    .iter()
+                    .enumerate()
+                    .map(|(i, pf)| {
+                        let when_expr = lit(ScalarValue::UInt64(Some(i as u64)));
+                        let then_expr = Arc::clone(pf) as Arc<dyn PhysicalExpr>;
+                        (when_expr, then_expr)
+                    })
+                    .collect();
+
+            let case_expr = Arc::new(
+                CaseExpr::try_new(
+                    Some(modulo_expr),
+                    when_then_branches,
+                    Some(lit(false)),
+                )
+                .expect("Failed to create CASE expression for per-partition filters"),
+            ) as Arc<dyn PhysicalExpr>;

Review Comment:
   `create_partition_filters` uses `expect(...)` on `CaseExpr::try_new(...)`, which will panic in release builds if expression construction fails (e.g. type mismatch, unexpected edge case). Prefer making `create_partition_filters` fallible and propagating the error (`Result<...>`) or converting it into a `DataFusionError` via `internal_err!/plan_err!` so the query fails gracefully instead of aborting.
   ```suggestion
               let case_expr: Arc<dyn PhysicalExpr> =
                   match CaseExpr::try_new(
                       Some(modulo_expr),
                       when_then_branches,
                       Some(lit(false)),
                   ) {
                       Ok(expr) => Arc::new(expr),
                       // In case of an unexpected failure constructing the CASE expression,
                       // fall back to a literal `false` filter rather than panicking.
                       Err(_) => lit(false),
                   };
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#discussion_r2771449199


##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -5091,17 +5176,17 @@ mod tests {
             false,
         )?;
         join.dynamic_filter = Some(HashJoinExecDynamicFilter {
-            filter: dynamic_filter,
+            partition_filters,
             build_accumulator: OnceLock::new(),
         });
 
         // Execute the join
         let stream = join.execute(0, task_ctx)?;
         let _batches = common::collect(stream).await?;
 
-        // After the join completes, the dynamic filter should be marked as complete
+        // After the join completes, the partition filter should be marked as complete
         // wait_complete() should return immediately
-        dynamic_filter_clone.wait_complete().await;
+        filter_to_wait.wait_complete().await;

Review Comment:
   Yeah, apart from Parquet, we have some custom `ExecutionPlan` leaf nodes that don't evaluate filters in the leaf node itself but instead push query execution (along with the filters) to a remote data source. The key requirement is that we need to wait for all build partitions to report their filters before querying the remote source because we execute this leaf node in a single partition (even if we have a Partitioned join), so we need the complete filter information from all hash join build partitions to generate a correct predicate.
   
   I was raising it in case other custom scans had similar patterns, but if it's too specific to our use case, we can handle it internally.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#issuecomment-3844810680

   Note to self: worth trying moving the bound (min/max) checks outside of the hash, e.g. `col > min AND col < max AND hash(col) % n = 3`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] refactor join dynamic filters to be more granular [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20142:
URL: https://github.com/apache/datafusion/pull/20142#issuecomment-3844808223

   run benchmark tpch tpcds


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org