You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "LiaCastaneda (via GitHub)" <gi...@apache.org> on 2025/11/26 10:39:08 UTC

[PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

LiaCastaneda opened a new pull request, #18938:
URL: https://github.com/apache/datafusion/pull/18938

   ## Which issue does this PR close?
   
   Closes https://github.com/apache/datafusion/issues/17527
   
   
   ## Rationale for this change
   
   Currently, DataFusion computes bounds for all queries that contain a HashJoinExec node whenever the option enable_dynamic_filter_pushdown is set to true (default). It might make sense to compute these bounds only when we explicitly know there is a consumer that will use them.
   
   ## What changes are included in this PR?
   
   As suggested in https://github.com/apache/datafusion/issues/17527#issuecomment-3576945224, this PR adds an is_used() method to DynamicFilterPhysicalExpr that checks if any consumers are holding a reference to the filter using Arc::strong_count(). 
   
   During filter pushdown, consumers that accept the filter and use it later in execution have to retain a reference to Arc<DynamicFilterPhysicalExpr>. For example, scan nodes like ParquetSource.
   
   ## Are these changes tested?
   
   I added a unit test in dynamic_filters.rs (test_is_used) that verifies the Arc reference counting behavior.
   Existing integration tests in datafusion/core/tests/physical_optimizer/filter_pushdown/mod.rs validate the end-to-end behavior. These tests verify that dynamic filters are computed and filled when consumers are present. 
   
   ## Are there any user-facing changes?
   
   new is_used() function
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3607964367

   I didn't had time to get back to this :(  I’ll be on vacation until next next week, so I won’t be able to look at this again until then. If anyone wants to take over feel free, otherwise, I’ll continue when I’m back!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#discussion_r2578967858


##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -1159,34 +1164,38 @@ impl ExecutionPlan for HashJoinExec {
         let right_child_self_filters = &child_pushdown_result.self_filters[1]; // We only push down filters to the right child
                                                                                // We expect 0 or 1 self filters
         if let Some(filter) = right_child_self_filters.first() {
-            // Note that we don't check PushdDownPredicate::discrimnant because even if nothing said
-            // "yes, I can fully evaluate this filter" things might still use it for statistics -> it's worth updating
-            let predicate = Arc::clone(&filter.predicate);
-            if let Ok(dynamic_filter) =
-                Arc::downcast::<DynamicFilterPhysicalExpr>(predicate)
-            {
-                // We successfully pushed down our self filter - we need to make a new node with the dynamic filter
-                let new_node = Arc::new(HashJoinExec {
-                    left: Arc::clone(&self.left),
-                    right: Arc::clone(&self.right),
-                    on: self.on.clone(),
-                    filter: self.filter.clone(),
-                    join_type: self.join_type,
-                    join_schema: Arc::clone(&self.join_schema),
-                    left_fut: Arc::clone(&self.left_fut),
-                    random_state: self.random_state.clone(),
-                    mode: self.mode,
-                    metrics: ExecutionPlanMetricsSet::new(),
-                    projection: self.projection.clone(),
-                    column_indices: self.column_indices.clone(),
-                    null_equality: self.null_equality,
-                    cache: self.cache.clone(),
-                    dynamic_filter: Some(HashJoinExecDynamicFilter {
-                        filter: dynamic_filter,
-                        build_accumulator: OnceLock::new(),
-                    }),
-                });
-                result = result.with_updated_node(new_node as Arc<dyn ExecutionPlan>);
+            // Only create the dynamic filter if the probe side will actually use it (Exact or Inexact).
+            // If it's Unsupported, don't compute the filter since it won't be used.
+            let will_be_used = !matches!(filter.discriminant, PushedDown::Unsupported);

Review Comment:
   I think these are both good reasons to make this API change. Maybe we can justify the change by doing what you are saying and skipping pushing down the entire hash table if it won't be used? But then again that is basically free... and where do bloom filters fall into this calculation? i.e. bloom filter pruning only works if HashJoinExec produces an InListExpr *and* the scan node is a Parquet node (or other format that supports bloom filters). That seems like an awful lot of complex coordination between the producer/consumer that is specific to each file (some may have bloom filters some don't) and the filter being pushed down (min/max vs. InList vs. Hash Table).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3598650556

   > I ask because I think this approach is what we actually want.
   
   Yeah, sorry for the confusion -- I switched the approach to exact/inexact/unsupported just as an alternative since it was getting really hard to get `is_used` fully right. I will open a separate PR for `is_used` to avoid confusion, I will attempt to solve the bug (or at least find out what's going on) between today and tomorrow. 
   
   > This is interesting. Could it indicate a bug?
   
   I took a look the other day, and the thing is that for some reason the strong count is not accounting for the leaf node that supposedly also holds a reference to `DynamicFilterPhysicalExpr`. πŸ€”  
   
   ```
   [partition 0] Arc count at execution: 1
   [partition 0] Is used: false
   [partition 1] Arc count at execution: 2
   ```
   
   The 1 -> 2 strong count is actually expected here because the first partition creates the SharedBuildAccumulator which also holds a reference to the DynamicFilterPhysicalExpr, but if the hash join + the leaf keep the ref (which is what we should be seeing) I would have expected to see something like:
   
   ```
   [partition 0] Arc count at execution: 2
   [partition 0] Is used: false
   [partition 1] Arc count at execution: 3
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3602010770

   πŸ€–: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   Comparing HEAD and lia_compute-dyn-filters-only-when-consumer-asks-for-it
   --------------------
   Benchmark clickbench_extended.json
   --------------------
   ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Query        ┃        HEAD ┃ lia_compute-dyn-filters-only-when-consumer-asks-for-it ┃    Change ┃
   ┑━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   β”‚ QQuery 0     β”‚  2767.67 ms β”‚                                             2740.62 ms β”‚ no change β”‚
   β”‚ QQuery 1     β”‚  1262.43 ms β”‚                                             1287.83 ms β”‚ no change β”‚
   β”‚ QQuery 2     β”‚  2465.37 ms β”‚                                             2422.75 ms β”‚ no change β”‚
   β”‚ QQuery 3     β”‚  1172.47 ms β”‚                                             1119.26 ms β”‚ no change β”‚
   β”‚ QQuery 4     β”‚  2364.16 ms β”‚                                             2343.32 ms β”‚ no change β”‚
   β”‚ QQuery 5     β”‚ 28802.93 ms β”‚                                            28731.60 ms β”‚ no change β”‚
   β”‚ QQuery 6     β”‚  4258.92 ms β”‚                                             4279.77 ms β”‚ no change β”‚
   β”‚ QQuery 7     β”‚  3588.21 ms β”‚                                             3490.75 ms β”‚ no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                                     ┃            ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                                                     β”‚ 46682.16ms β”‚
   β”‚ Total Time (lia_compute-dyn-filters-only-when-consumer-asks-for-it)   β”‚ 46415.89ms β”‚
   β”‚ Average Time (HEAD)                                                   β”‚  5835.27ms β”‚
   β”‚ Average Time (lia_compute-dyn-filters-only-when-consumer-asks-for-it) β”‚  5801.99ms β”‚
   β”‚ Queries Faster                                                        β”‚          0 β”‚
   β”‚ Queries Slower                                                        β”‚          0 β”‚
   β”‚ Queries with No Change                                                β”‚          8 β”‚
   β”‚ Queries with Failure                                                  β”‚          0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query        ┃        HEAD ┃ lia_compute-dyn-filters-only-when-consumer-asks-for-it ┃        Change ┃
   ┑━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   β”‚ QQuery 0     β”‚     2.55 ms β”‚                                                2.68 ms β”‚     no change β”‚
   β”‚ QQuery 1     β”‚    52.02 ms β”‚                                               50.21 ms β”‚     no change β”‚
   β”‚ QQuery 2     β”‚   136.76 ms β”‚                                              137.00 ms β”‚     no change β”‚
   β”‚ QQuery 3     β”‚   169.62 ms β”‚                                              167.50 ms β”‚     no change β”‚
   β”‚ QQuery 4     β”‚  1187.17 ms β”‚                                             1161.64 ms β”‚     no change β”‚
   β”‚ QQuery 5     β”‚  1590.44 ms β”‚                                             1546.91 ms β”‚     no change β”‚
   β”‚ QQuery 6     β”‚     2.32 ms β”‚                                                2.12 ms β”‚ +1.09x faster β”‚
   β”‚ QQuery 7     β”‚    56.54 ms β”‚                                               54.82 ms β”‚     no change β”‚
   β”‚ QQuery 8     β”‚  1511.29 ms β”‚                                             1509.88 ms β”‚     no change β”‚
   β”‚ QQuery 9     β”‚  1990.88 ms β”‚                                             1955.38 ms β”‚     no change β”‚
   β”‚ QQuery 10    β”‚   403.54 ms β”‚                                              409.87 ms β”‚     no change β”‚
   β”‚ QQuery 11    β”‚   451.48 ms β”‚                                              457.59 ms β”‚     no change β”‚
   β”‚ QQuery 12    β”‚  1470.94 ms β”‚                                             1485.98 ms β”‚     no change β”‚
   β”‚ QQuery 13    β”‚  2217.06 ms β”‚                                             2133.54 ms β”‚     no change β”‚
   β”‚ QQuery 14    β”‚  1337.52 ms β”‚                                             1329.19 ms β”‚     no change β”‚
   β”‚ QQuery 15    β”‚  1307.39 ms β”‚                                             1301.05 ms β”‚     no change β”‚
   β”‚ QQuery 16    β”‚  2752.91 ms β”‚                                             2740.93 ms β”‚     no change β”‚
   β”‚ QQuery 17    β”‚  2738.79 ms β”‚                                             2701.23 ms β”‚     no change β”‚
   β”‚ QQuery 18    β”‚  5113.70 ms β”‚                                             5073.84 ms β”‚     no change β”‚
   β”‚ QQuery 19    β”‚   130.34 ms β”‚                                              130.32 ms β”‚     no change β”‚
   β”‚ QQuery 20    β”‚  2059.88 ms β”‚                                             2002.18 ms β”‚     no change β”‚
   β”‚ QQuery 21    β”‚  2359.42 ms β”‚                                             2310.55 ms β”‚     no change β”‚
   β”‚ QQuery 22    β”‚  4012.46 ms β”‚                                             3913.87 ms β”‚     no change β”‚
   β”‚ QQuery 23    β”‚ 13467.65 ms β”‚                                            13188.10 ms β”‚     no change β”‚
   β”‚ QQuery 24    β”‚   236.69 ms β”‚                                              234.30 ms β”‚     no change β”‚
   β”‚ QQuery 25    β”‚   499.69 ms β”‚                                              491.81 ms β”‚     no change β”‚
   β”‚ QQuery 26    β”‚   234.39 ms β”‚                                              218.38 ms β”‚ +1.07x faster β”‚
   β”‚ QQuery 27    β”‚  2966.96 ms β”‚                                             2828.76 ms β”‚     no change β”‚
   β”‚ QQuery 28    β”‚ 23816.85 ms β”‚                                            23563.87 ms β”‚     no change β”‚
   β”‚ QQuery 29    β”‚   943.64 ms β”‚                                              947.57 ms β”‚     no change β”‚
   β”‚ QQuery 30    β”‚  1386.29 ms β”‚                                             1399.26 ms β”‚     no change β”‚
   β”‚ QQuery 31    β”‚  1412.73 ms β”‚                                             1418.85 ms β”‚     no change β”‚
   β”‚ QQuery 32    β”‚  4623.55 ms β”‚                                             4803.31 ms β”‚     no change β”‚
   β”‚ QQuery 33    β”‚  5898.12 ms β”‚                                             6005.81 ms β”‚     no change β”‚
   β”‚ QQuery 34    β”‚  6020.30 ms β”‚                                             6059.38 ms β”‚     no change β”‚
   β”‚ QQuery 35    β”‚  1960.47 ms β”‚                                             1949.46 ms β”‚     no change β”‚
   β”‚ QQuery 36    β”‚   122.08 ms β”‚                                              121.00 ms β”‚     no change β”‚
   β”‚ QQuery 37    β”‚    51.18 ms β”‚                                               55.14 ms β”‚  1.08x slower β”‚
   β”‚ QQuery 38    β”‚   120.04 ms β”‚                                              122.63 ms β”‚     no change β”‚
   β”‚ QQuery 39    β”‚   198.46 ms β”‚                                              199.93 ms β”‚     no change β”‚
   β”‚ QQuery 40    β”‚    45.03 ms β”‚                                               43.30 ms β”‚     no change β”‚
   β”‚ QQuery 41    β”‚    41.59 ms β”‚                                               39.01 ms β”‚ +1.07x faster β”‚
   β”‚ QQuery 42    β”‚    33.88 ms β”‚                                               32.37 ms β”‚     no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                                     ┃            ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                                                     β”‚ 97134.62ms β”‚
   β”‚ Total Time (lia_compute-dyn-filters-only-when-consumer-asks-for-it)   β”‚ 96300.53ms β”‚
   β”‚ Average Time (HEAD)                                                   β”‚  2258.94ms β”‚
   β”‚ Average Time (lia_compute-dyn-filters-only-when-consumer-asks-for-it) β”‚  2239.55ms β”‚
   β”‚ Queries Faster                                                        β”‚          3 β”‚
   β”‚ Queries Slower                                                        β”‚          1 β”‚
   β”‚ Queries with No Change                                                β”‚         39 β”‚
   β”‚ Queries with Failure                                                  β”‚          0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   --------------------
   Benchmark tpch_mem_sf1.json
   --------------------
   ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query        ┃      HEAD ┃ lia_compute-dyn-filters-only-when-consumer-asks-for-it ┃        Change ┃
   ┑━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   β”‚ QQuery 1     β”‚ 131.73 ms β”‚                                              140.41 ms β”‚  1.07x slower β”‚
   β”‚ QQuery 2     β”‚  27.82 ms β”‚                                               28.02 ms β”‚     no change β”‚
   β”‚ QQuery 3     β”‚  39.56 ms β”‚                                               35.74 ms β”‚ +1.11x faster β”‚
   β”‚ QQuery 4     β”‚  29.95 ms β”‚                                               29.67 ms β”‚     no change β”‚
   β”‚ QQuery 5     β”‚  89.46 ms β”‚                                               87.23 ms β”‚     no change β”‚
   β”‚ QQuery 6     β”‚  19.88 ms β”‚                                               19.53 ms β”‚     no change β”‚
   β”‚ QQuery 7     β”‚ 228.13 ms β”‚                                              228.75 ms β”‚     no change β”‚
   β”‚ QQuery 8     β”‚  36.75 ms β”‚                                               35.39 ms β”‚     no change β”‚
   β”‚ QQuery 9     β”‚ 109.11 ms β”‚                                              104.25 ms β”‚     no change β”‚
   β”‚ QQuery 10    β”‚  68.04 ms β”‚                                               63.60 ms β”‚ +1.07x faster β”‚
   β”‚ QQuery 11    β”‚  18.48 ms β”‚                                               18.83 ms β”‚     no change β”‚
   β”‚ QQuery 12    β”‚  51.29 ms β”‚                                               51.85 ms β”‚     no change β”‚
   β”‚ QQuery 13    β”‚  47.86 ms β”‚                                               48.06 ms β”‚     no change β”‚
   β”‚ QQuery 14    β”‚  14.17 ms β”‚                                               13.95 ms β”‚     no change β”‚
   β”‚ QQuery 15    β”‚  24.99 ms β”‚                                               24.89 ms β”‚     no change β”‚
   β”‚ QQuery 16    β”‚  25.18 ms β”‚                                               25.17 ms β”‚     no change β”‚
   β”‚ QQuery 17    β”‚ 155.49 ms β”‚                                              156.26 ms β”‚     no change β”‚
   β”‚ QQuery 18    β”‚ 285.00 ms β”‚                                              277.43 ms β”‚     no change β”‚
   β”‚ QQuery 19    β”‚  39.68 ms β”‚                                               37.68 ms β”‚ +1.05x faster β”‚
   β”‚ QQuery 20    β”‚  51.35 ms β”‚                                               50.76 ms β”‚     no change β”‚
   β”‚ QQuery 21    β”‚ 330.72 ms β”‚                                              333.45 ms β”‚     no change β”‚
   β”‚ QQuery 22    β”‚  18.13 ms β”‚                                               17.88 ms β”‚     no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                                     ┃           ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                                                     β”‚ 1842.76ms β”‚
   β”‚ Total Time (lia_compute-dyn-filters-only-when-consumer-asks-for-it)   β”‚ 1828.79ms β”‚
   β”‚ Average Time (HEAD)                                                   β”‚   83.76ms β”‚
   β”‚ Average Time (lia_compute-dyn-filters-only-when-consumer-asks-for-it) β”‚   83.13ms β”‚
   β”‚ Queries Faster                                                        β”‚         3 β”‚
   β”‚ Queries Slower                                                        β”‚         1 β”‚
   β”‚ Queries with No Change                                                β”‚        18 β”‚
   β”‚ Queries with Failure                                                  β”‚         0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda closed pull request #18938: Compute Dynamic Filters only when a consumer supports them
URL: https://github.com/apache/datafusion/pull/18938


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3580845809

   Something funky is going on here with the Arc count, will take a look...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3601753252

   show benchmark queue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3598215540

   πŸ€–: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   Comparing HEAD and lia_compute-dyn-filters-only-when-consumer-asks-for-it
   --------------------
   Benchmark clickbench_extended.json
   --------------------
   ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Query        ┃        HEAD ┃ lia_compute-dyn-filters-only-when-consumer-asks-for-it ┃    Change ┃
   ┑━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   β”‚ QQuery 0     β”‚  2697.10 ms β”‚                                             2748.66 ms β”‚ no change β”‚
   β”‚ QQuery 1     β”‚  1316.46 ms β”‚                                             1310.41 ms β”‚ no change β”‚
   β”‚ QQuery 2     β”‚  2484.15 ms β”‚                                             2498.55 ms β”‚ no change β”‚
   β”‚ QQuery 3     β”‚  1116.93 ms β”‚                                             1110.21 ms β”‚ no change β”‚
   β”‚ QQuery 4     β”‚  2371.11 ms β”‚                                             2321.44 ms β”‚ no change β”‚
   β”‚ QQuery 5     β”‚ 28306.32 ms β”‚                                            28476.49 ms β”‚ no change β”‚
   β”‚ QQuery 6     β”‚  4230.39 ms β”‚                                             4237.43 ms β”‚ no change β”‚
   β”‚ QQuery 7     β”‚  3647.99 ms β”‚                                             3615.26 ms β”‚ no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                                     ┃            ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                                                     β”‚ 46170.45ms β”‚
   β”‚ Total Time (lia_compute-dyn-filters-only-when-consumer-asks-for-it)   β”‚ 46318.45ms β”‚
   β”‚ Average Time (HEAD)                                                   β”‚  5771.31ms β”‚
   β”‚ Average Time (lia_compute-dyn-filters-only-when-consumer-asks-for-it) β”‚  5789.81ms β”‚
   β”‚ Queries Faster                                                        β”‚          0 β”‚
   β”‚ Queries Slower                                                        β”‚          0 β”‚
   β”‚ Queries with No Change                                                β”‚          8 β”‚
   β”‚ Queries with Failure                                                  β”‚          0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   --------------------
   Benchmark clickbench_partitioned.json
   --------------------
   ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query        ┃        HEAD ┃ lia_compute-dyn-filters-only-when-consumer-asks-for-it ┃        Change ┃
   ┑━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   β”‚ QQuery 0     β”‚     2.53 ms β”‚                                                2.21 ms β”‚ +1.15x faster β”‚
   β”‚ QQuery 1     β”‚    50.65 ms β”‚                                               48.48 ms β”‚     no change β”‚
   β”‚ QQuery 2     β”‚   138.85 ms β”‚                                              137.64 ms β”‚     no change β”‚
   β”‚ QQuery 3     β”‚   166.42 ms β”‚                                              167.48 ms β”‚     no change β”‚
   β”‚ QQuery 4     β”‚  1074.24 ms β”‚                                             1107.96 ms β”‚     no change β”‚
   β”‚ QQuery 5     β”‚  1500.86 ms β”‚                                             1505.22 ms β”‚     no change β”‚
   β”‚ QQuery 6     β”‚     2.19 ms β”‚                                                2.14 ms β”‚     no change β”‚
   β”‚ QQuery 7     β”‚    55.99 ms β”‚                                               55.80 ms β”‚     no change β”‚
   β”‚ QQuery 8     β”‚  1433.46 ms β”‚                                             1438.73 ms β”‚     no change β”‚
   β”‚ QQuery 9     β”‚  1834.71 ms β”‚                                             1855.08 ms β”‚     no change β”‚
   β”‚ QQuery 10    β”‚   396.37 ms β”‚                                              397.92 ms β”‚     no change β”‚
   β”‚ QQuery 11    β”‚   445.08 ms β”‚                                              450.27 ms β”‚     no change β”‚
   β”‚ QQuery 12    β”‚  1337.56 ms β”‚                                             1368.91 ms β”‚     no change β”‚
   β”‚ QQuery 13    β”‚  2159.17 ms β”‚                                             2141.45 ms β”‚     no change β”‚
   β”‚ QQuery 14    β”‚  1253.87 ms β”‚                                             1285.80 ms β”‚     no change β”‚
   β”‚ QQuery 15    β”‚  1214.01 ms β”‚                                             1227.89 ms β”‚     no change β”‚
   β”‚ QQuery 16    β”‚  2713.52 ms β”‚                                             2727.51 ms β”‚     no change β”‚
   β”‚ QQuery 17    β”‚  2672.03 ms β”‚                                             2696.53 ms β”‚     no change β”‚
   β”‚ QQuery 18    β”‚  5312.51 ms β”‚                                             5035.52 ms β”‚ +1.06x faster β”‚
   β”‚ QQuery 19    β”‚   130.21 ms β”‚                                              126.79 ms β”‚     no change β”‚
   β”‚ QQuery 20    β”‚  2016.14 ms β”‚                                             1991.80 ms β”‚     no change β”‚
   β”‚ QQuery 21    β”‚  2342.35 ms β”‚                                             2312.08 ms β”‚     no change β”‚
   β”‚ QQuery 22    β”‚  3948.29 ms β”‚                                             3949.44 ms β”‚     no change β”‚
   β”‚ QQuery 23    β”‚ 14706.89 ms β”‚                                            13287.83 ms β”‚ +1.11x faster β”‚
   β”‚ QQuery 24    β”‚   231.24 ms β”‚                                              218.60 ms β”‚ +1.06x faster β”‚
   β”‚ QQuery 25    β”‚   485.79 ms β”‚                                              489.69 ms β”‚     no change β”‚
   β”‚ QQuery 26    β”‚   245.70 ms β”‚                                              217.96 ms β”‚ +1.13x faster β”‚
   β”‚ QQuery 27    β”‚  2904.01 ms β”‚                                             2833.98 ms β”‚     no change β”‚
   β”‚ QQuery 28    β”‚ 23753.40 ms β”‚                                            23583.56 ms β”‚     no change β”‚
   β”‚ QQuery 29    β”‚   980.90 ms β”‚                                              973.83 ms β”‚     no change β”‚
   β”‚ QQuery 30    β”‚  1374.88 ms β”‚                                             1339.59 ms β”‚     no change β”‚
   β”‚ QQuery 31    β”‚  1408.58 ms β”‚                                             1402.35 ms β”‚     no change β”‚
   β”‚ QQuery 32    β”‚  4834.01 ms β”‚                                             5151.11 ms β”‚  1.07x slower β”‚
   β”‚ QQuery 33    β”‚  5903.63 ms β”‚                                             5954.64 ms β”‚     no change β”‚
   β”‚ QQuery 34    β”‚  6095.62 ms β”‚                                             6197.45 ms β”‚     no change β”‚
   β”‚ QQuery 35    β”‚  1915.96 ms β”‚                                             1875.74 ms β”‚     no change β”‚
   β”‚ QQuery 36    β”‚   118.10 ms β”‚                                              120.20 ms β”‚     no change β”‚
   β”‚ QQuery 37    β”‚    52.24 ms β”‚                                               51.90 ms β”‚     no change β”‚
   β”‚ QQuery 38    β”‚   121.59 ms β”‚                                              119.37 ms β”‚     no change β”‚
   β”‚ QQuery 39    β”‚   198.20 ms β”‚                                              198.99 ms β”‚     no change β”‚
   β”‚ QQuery 40    β”‚    42.91 ms β”‚                                               41.56 ms β”‚     no change β”‚
   β”‚ QQuery 41    β”‚    40.53 ms β”‚                                               40.88 ms β”‚     no change β”‚
   β”‚ QQuery 42    β”‚    33.56 ms β”‚                                               32.87 ms β”‚     no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
   ┃ Benchmark Summary                                                     ┃            ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                                                     β”‚ 97648.75ms β”‚
   β”‚ Total Time (lia_compute-dyn-filters-only-when-consumer-asks-for-it)   β”‚ 96164.73ms β”‚
   β”‚ Average Time (HEAD)                                                   β”‚  2270.90ms β”‚
   β”‚ Average Time (lia_compute-dyn-filters-only-when-consumer-asks-for-it) β”‚  2236.39ms β”‚
   β”‚ Queries Faster                                                        β”‚          5 β”‚
   β”‚ Queries Slower                                                        β”‚          1 β”‚
   β”‚ Queries with No Change                                                β”‚         37 β”‚
   β”‚ Queries with Failure                                                  β”‚          0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   --------------------
   Benchmark tpch_mem_sf1.json
   --------------------
   ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query        ┃      HEAD ┃ lia_compute-dyn-filters-only-when-consumer-asks-for-it ┃        Change ┃
   ┑━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   β”‚ QQuery 1     β”‚ 128.18 ms β”‚                                              139.76 ms β”‚  1.09x slower β”‚
   β”‚ QQuery 2     β”‚  27.45 ms β”‚                                               26.80 ms β”‚     no change β”‚
   β”‚ QQuery 3     β”‚  38.94 ms β”‚                                               36.01 ms β”‚ +1.08x faster β”‚
   β”‚ QQuery 4     β”‚  28.49 ms β”‚                                               28.88 ms β”‚     no change β”‚
   β”‚ QQuery 5     β”‚  87.67 ms β”‚                                               86.86 ms β”‚     no change β”‚
   β”‚ QQuery 6     β”‚  19.14 ms β”‚                                               19.34 ms β”‚     no change β”‚
   β”‚ QQuery 7     β”‚ 216.95 ms β”‚                                              224.91 ms β”‚     no change β”‚
   β”‚ QQuery 8     β”‚  34.24 ms β”‚                                               30.97 ms β”‚ +1.11x faster β”‚
   β”‚ QQuery 9     β”‚ 105.71 ms β”‚                                              101.77 ms β”‚     no change β”‚
   β”‚ QQuery 10    β”‚  63.70 ms β”‚                                               62.73 ms β”‚     no change β”‚
   β”‚ QQuery 11    β”‚  19.00 ms β”‚                                               18.50 ms β”‚     no change β”‚
   β”‚ QQuery 12    β”‚  53.13 ms β”‚                                               51.61 ms β”‚     no change β”‚
   β”‚ QQuery 13    β”‚  44.97 ms β”‚                                               48.78 ms β”‚  1.08x slower β”‚
   β”‚ QQuery 14    β”‚  14.27 ms β”‚                                               13.56 ms β”‚     no change β”‚
   β”‚ QQuery 15    β”‚  24.14 ms β”‚                                               24.30 ms β”‚     no change β”‚
   β”‚ QQuery 16    β”‚  25.07 ms β”‚                                               24.75 ms β”‚     no change β”‚
   β”‚ QQuery 17    β”‚ 150.74 ms β”‚                                              152.84 ms β”‚     no change β”‚
   β”‚ QQuery 18    β”‚ 286.35 ms β”‚                                              281.81 ms β”‚     no change β”‚
   β”‚ QQuery 19    β”‚  39.98 ms β”‚                                               38.59 ms β”‚     no change β”‚
   β”‚ QQuery 20    β”‚  49.11 ms β”‚                                               49.62 ms β”‚     no change β”‚
   β”‚ QQuery 21    β”‚ 325.29 ms β”‚                                              314.30 ms β”‚     no change β”‚
   β”‚ QQuery 22    β”‚  17.56 ms β”‚                                               17.61 ms β”‚     no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                                     ┃           ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                                                     β”‚ 1800.07ms β”‚
   β”‚ Total Time (lia_compute-dyn-filters-only-when-consumer-asks-for-it)   β”‚ 1794.29ms β”‚
   β”‚ Average Time (HEAD)                                                   β”‚   81.82ms β”‚
   β”‚ Average Time (lia_compute-dyn-filters-only-when-consumer-asks-for-it) β”‚   81.56ms β”‚
   β”‚ Queries Faster                                                        β”‚         2 β”‚
   β”‚ Queries Slower                                                        β”‚         2 β”‚
   β”‚ Queries with No Change                                                β”‚        18 β”‚
   β”‚ Queries with Failure                                                  β”‚         0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#discussion_r2578190450


##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -910,6 +910,11 @@ impl ExecutionPlan for HashJoinExec {
              consider using CoalescePartitionsExec or the EnforceDistribution rule"
         );
 
+        // Only enable dynamic filter pushdown if:
+        // - The session config enables dynamic filter pushdown
+        // - A dynamic filter exists
+        // - At least one consumer is holding a reference to it, this avoids expensive filter

Review Comment:
   This comment is outdated



##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -1159,34 +1164,38 @@ impl ExecutionPlan for HashJoinExec {
         let right_child_self_filters = &child_pushdown_result.self_filters[1]; // We only push down filters to the right child
                                                                                // We expect 0 or 1 self filters
         if let Some(filter) = right_child_self_filters.first() {
-            // Note that we don't check PushdDownPredicate::discrimnant because even if nothing said
-            // "yes, I can fully evaluate this filter" things might still use it for statistics -> it's worth updating
-            let predicate = Arc::clone(&filter.predicate);
-            if let Ok(dynamic_filter) =
-                Arc::downcast::<DynamicFilterPhysicalExpr>(predicate)
-            {
-                // We successfully pushed down our self filter - we need to make a new node with the dynamic filter
-                let new_node = Arc::new(HashJoinExec {
-                    left: Arc::clone(&self.left),
-                    right: Arc::clone(&self.right),
-                    on: self.on.clone(),
-                    filter: self.filter.clone(),
-                    join_type: self.join_type,
-                    join_schema: Arc::clone(&self.join_schema),
-                    left_fut: Arc::clone(&self.left_fut),
-                    random_state: self.random_state.clone(),
-                    mode: self.mode,
-                    metrics: ExecutionPlanMetricsSet::new(),
-                    projection: self.projection.clone(),
-                    column_indices: self.column_indices.clone(),
-                    null_equality: self.null_equality,
-                    cache: self.cache.clone(),
-                    dynamic_filter: Some(HashJoinExecDynamicFilter {
-                        filter: dynamic_filter,
-                        build_accumulator: OnceLock::new(),
-                    }),
-                });
-                result = result.with_updated_node(new_node as Arc<dyn ExecutionPlan>);
+            // Only create the dynamic filter if the probe side will actually use it (Exact or Inexact).
+            // If it's Unsupported, don't compute the filter since it won't be used.
+            let will_be_used = !matches!(filter.discriminant, PushedDown::Unsupported);

Review Comment:
   If this is the case, don't we end up in the same place as `Yes/No`? I.e. this change only seems helpful if we did something like "only create the filter if the child said `Exact`".



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3598974879

   
   run benchmarks
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#discussion_r2580405974


##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -1159,34 +1164,38 @@ impl ExecutionPlan for HashJoinExec {
         let right_child_self_filters = &child_pushdown_result.self_filters[1]; // We only push down filters to the right child
                                                                                // We expect 0 or 1 self filters
         if let Some(filter) = right_child_self_filters.first() {
-            // Note that we don't check PushdDownPredicate::discrimnant because even if nothing said
-            // "yes, I can fully evaluate this filter" things might still use it for statistics -> it's worth updating
-            let predicate = Arc::clone(&filter.predicate);
-            if let Ok(dynamic_filter) =
-                Arc::downcast::<DynamicFilterPhysicalExpr>(predicate)
-            {
-                // We successfully pushed down our self filter - we need to make a new node with the dynamic filter
-                let new_node = Arc::new(HashJoinExec {
-                    left: Arc::clone(&self.left),
-                    right: Arc::clone(&self.right),
-                    on: self.on.clone(),
-                    filter: self.filter.clone(),
-                    join_type: self.join_type,
-                    join_schema: Arc::clone(&self.join_schema),
-                    left_fut: Arc::clone(&self.left_fut),
-                    random_state: self.random_state.clone(),
-                    mode: self.mode,
-                    metrics: ExecutionPlanMetricsSet::new(),
-                    projection: self.projection.clone(),
-                    column_indices: self.column_indices.clone(),
-                    null_equality: self.null_equality,
-                    cache: self.cache.clone(),
-                    dynamic_filter: Some(HashJoinExecDynamicFilter {
-                        filter: dynamic_filter,
-                        build_accumulator: OnceLock::new(),
-                    }),
-                });
-                result = result.with_updated_node(new_node as Arc<dyn ExecutionPlan>);
+            // Only create the dynamic filter if the probe side will actually use it (Exact or Inexact).
+            // If it's Unsupported, don't compute the filter since it won't be used.
+            let will_be_used = !matches!(filter.discriminant, PushedDown::Unsupported);

Review Comment:
   Yeah, it seems like it adds complexity to the β€œwhich filter to use” decision. Maybe the only clear use case is:
   
   > it lets scans say "I can't use this at all" (Unsupported), so we can skip computing filters entirely if stats prunning is not supported eitheri think it lets scans say "I can't use this at all" (Unsupported), so we can skip computing filters entirely if stats prunning is not supported either
   
   I was also thinking, what if we just let consumers communicate what kinds of filters they support, and the producer only adjusts that decision based on memory or row-count limits? Or would that be an anti-pattern? In any case, that still wouldn’t let the producer understand the purpose of the dynamic filters (if for stats or row level filtering)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3628678941

   Got it -- thanks -- I'll mark this one as a draft then so it is more clear it is waiting on additional input


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#discussion_r2564502734


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -294,6 +294,16 @@ impl DynamicFilterPhysicalExpr {
             .await;
     }
 
+    /// Check if this dynamic filter is being actively used by any consumers.
+    ///
+    /// Returns `true` if there are references beyond the producer (e.g., the HashJoinExec
+    /// that created the filter). This is useful to avoid computing expensive filter
+    /// expressions when no consumer will actually use them.
+    pub fn is_used(self: &Arc<Self>) -> bool {
+        // Strong count > 1 means at least one consumer is holding a reference beyond the producer.
+        Arc::strong_count(self) > 1
+    }

Review Comment:
   I'm not really sure how to test a condition where is_used() returns false without adding too much machinery or making the `dynamic_filter` attribute from HashJoin public which would make it easy to mess with the Arc reference count. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3599038336

   @alamb I will attempt to get right the other approach (`is_used`) first, as @adriangb maybe we shouldn't modify the API unless we have an clear need for a third (stat only) discriminant. Lets not merge this yet πŸ˜„ 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3696469893

   πŸ‘‹ I think I found the issue, in short, we should check the strong count of the inner struct not the DynamicFilterPhysicalExpr πŸ€¦β€β™€οΈ since sometimes new DynamicFilterPhysicalExpr are created but still keeping the same inner struct reference. I will push the updated commit in the same PR in a moment


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#discussion_r2564582722


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -294,6 +294,16 @@ impl DynamicFilterPhysicalExpr {
             .await;
     }
 
+    /// Check if this dynamic filter is being actively used by any consumers.
+    ///
+    /// Returns `true` if there are references beyond the producer (e.g., the HashJoinExec
+    /// that created the filter). This is useful to avoid computing expensive filter
+    /// expressions when no consumer will actually use them.
+    pub fn is_used(self: &Arc<Self>) -> bool {
+        // Strong count > 1 means at least one consumer is holding a reference beyond the producer.
+        Arc::strong_count(self) > 1
+    }

Review Comment:
   Can’t you just make a new DynamicFilterPhysicalExpr and check that is_used is False?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3628286336

   The current status from my perspective is that I pushed back on changing the 2 state logic of pushdown to 3 stats because (1) it generates a lot of API churn and (2) it's not clear that it covers all cases e.g. does bloom filters count as stats pruning or not, etc.
   
   An approach that only targets dynamic filters and answers the question "is someone keeping a reference to this" at least resolves the case where no one is keeping a reference to the filter and thus it can't possibly be used, while making the public API changes much smaller.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3597911092

   πŸ€– `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing lia/compute-dyn-filters-only-when-consumer-asks-for-it (728fde92251c23e1124f9bd29a9e9cdc395a90a6) to fb14d7cb1a5eea5b2b2999c1213493d6385447c5 [diff](https://github.com/apache/datafusion/compare/fb14d7cb1a5eea5b2b2999c1213493d6385447c5..728fde92251c23e1124f9bd29a9e9cdc395a90a6) using:  tpch_mem clickbench_partitioned clickbench_extended
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on code in PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#discussion_r2578637670


##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -1159,34 +1164,38 @@ impl ExecutionPlan for HashJoinExec {
         let right_child_self_filters = &child_pushdown_result.self_filters[1]; // We only push down filters to the right child
                                                                                // We expect 0 or 1 self filters
         if let Some(filter) = right_child_self_filters.first() {
-            // Note that we don't check PushdDownPredicate::discrimnant because even if nothing said
-            // "yes, I can fully evaluate this filter" things might still use it for statistics -> it's worth updating
-            let predicate = Arc::clone(&filter.predicate);
-            if let Ok(dynamic_filter) =
-                Arc::downcast::<DynamicFilterPhysicalExpr>(predicate)
-            {
-                // We successfully pushed down our self filter - we need to make a new node with the dynamic filter
-                let new_node = Arc::new(HashJoinExec {
-                    left: Arc::clone(&self.left),
-                    right: Arc::clone(&self.right),
-                    on: self.on.clone(),
-                    filter: self.filter.clone(),
-                    join_type: self.join_type,
-                    join_schema: Arc::clone(&self.join_schema),
-                    left_fut: Arc::clone(&self.left_fut),
-                    random_state: self.random_state.clone(),
-                    mode: self.mode,
-                    metrics: ExecutionPlanMetricsSet::new(),
-                    projection: self.projection.clone(),
-                    column_indices: self.column_indices.clone(),
-                    null_equality: self.null_equality,
-                    cache: self.cache.clone(),
-                    dynamic_filter: Some(HashJoinExecDynamicFilter {
-                        filter: dynamic_filter,
-                        build_accumulator: OnceLock::new(),
-                    }),
-                });
-                result = result.with_updated_node(new_node as Arc<dyn ExecutionPlan>);
+            // Only create the dynamic filter if the probe side will actually use it (Exact or Inexact).
+            // If it's Unsupported, don't compute the filter since it won't be used.
+            let will_be_used = !matches!(filter.discriminant, PushedDown::Unsupported);

Review Comment:
   i think it lets scans say "I can't use this at all" (Unsupported), so we can skip computing filters entirely if stats prunning is not supported either - the Yes/No system had no way to express that: if we had stats pruning with the filters, it would fall under the `No` discriminant, but we would still need them. I'm also thinking: if we know a scan will only use the filter for stats pruning (Inexact), maybe would it make sense to compute just the min/max bounds instead of both IN LIST and bounds?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3601752366

   πŸ€– `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing lia/compute-dyn-filters-only-when-consumer-asks-for-it (cb749ada94e165c0c503d1e66d302f9a47e7c75c) to fb14d7cb1a5eea5b2b2999c1213493d6385447c5 [diff](https://github.com/apache/datafusion/compare/fb14d7cb1a5eea5b2b2999c1213493d6385447c5..cb749ada94e165c0c503d1e66d302f9a47e7c75c) using:  tpch_mem clickbench_partitioned clickbench_extended
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3601753480

   πŸ€– Hi @alamb, you asked to view the benchmark queue (https://github.com/apache/datafusion/pull/18938#issuecomment-3601753252).
   
   | Job | User | Benchmarks | Comment |
   | --- | --- | --- | --- |
   | `18938_3598974879.sh` | alamb | default | `https://github.com/apache/datafusion/pull/18938#issuecomment-3598974879` |
   | `18972_3600358896.sh` | Dandandan | tpch_mem | `https://github.com/apache/datafusion/pull/18972#issuecomment-3600358896` |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3597335533

   run benchmarks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#discussion_r2564584138


##########
datafusion/physical-expr/src/expressions/dynamic_filters.rs:
##########
@@ -294,6 +294,16 @@ impl DynamicFilterPhysicalExpr {
             .await;
     }
 
+    /// Check if this dynamic filter is being actively used by any consumers.
+    ///
+    /// Returns `true` if there are references beyond the producer (e.g., the HashJoinExec
+    /// that created the filter). This is useful to avoid computing expensive filter
+    /// expressions when no consumer will actually use them.
+    pub fn is_used(self: &Arc<Self>) -> bool {
+        // Strong count > 1 means at least one consumer is holding a reference beyond the producer.
+        Arc::strong_count(self) > 1
+    }

Review Comment:
   Looks like you have a test already?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3628264936

   I think I lost track of the status of this PR -- can it be merged as is (with potential improvements as a follow on PR) or does it need more work prior to merge?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3701744117

   closing this PR in favour of https://github.com/apache/datafusion/pull/19501


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3701748470

   closed in favour of https://github.com/apache/datafusion/pull/19546


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#discussion_r2577655278


##########
datafusion/physical-plan/src/filter_pushdown.rs:
##########
@@ -101,46 +101,84 @@ impl PushedDownPredicate {
         self.predicate
     }
 
-    /// Create a new [`PushedDownPredicate`] with supported pushdown.
+    /// Create a new [`PushedDownPredicate`] with exact pushdown.
+    /// The child will apply the filter exactly as FilterExec would.
+    pub fn exact(predicate: Arc<dyn PhysicalExpr>) -> Self {
+        Self {
+            discriminant: PushedDown::Exact,
+            predicate,
+        }
+    }
+
+    /// Create a new [`PushedDownPredicate`] with supported pushdown (exact filtering).
+    /// This is an alias for `exact()` for backward compatibility.
     pub fn supported(predicate: Arc<dyn PhysicalExpr>) -> Self {
+        Self::exact(predicate)
+    }
+
+    /// Create a new [`PushedDownPredicate`] with inexact pushdown.
+    /// The child will use the filter for stats/partial pruning but not exact filtering.
+    pub fn inexact(predicate: Arc<dyn PhysicalExpr>) -> Self {
         Self {
-            discriminant: PushedDown::Yes,
+            discriminant: PushedDown::Inexact,
             predicate,
         }
     }
 
     /// Create a new [`PushedDownPredicate`] with unsupported pushdown.
     pub fn unsupported(predicate: Arc<dyn PhysicalExpr>) -> Self {
         Self {
-            discriminant: PushedDown::No,
+            discriminant: PushedDown::Unsupported,
             predicate,
         }
     }
 }
 
 /// Discriminant for the result of pushing down a filter into a child node.
-#[derive(Debug, Clone, Copy)]
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
 pub enum PushedDown {

Review Comment:
   Since this is a public enum ([doc link](https://docs.rs/datafusion/latest/datafusion/physical_plan/filter_pushdown/enum.PushedDown.html)) I think this will be an API change (I marked the PR as such)
   
   
   Can you please add a note to the DataFusion 52 upgrade guide explaining how people need to update their existing code (e.g. if they used to return `Yes` or `No` what should they return after this change)?
   
   https://github.com/apache/datafusion/blob/9f725d9c7064813cda0de0f87d115354b68d76e6/docs/source/library-user-guide/upgrading.md#L22-L21



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3696479821

   Actually, I think I will open a new PR so the other approach remains in this PR in case it comes handy in the future


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Compute Dynamic Filters only when a consumer supports them [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #18938:
URL: https://github.com/apache/datafusion/pull/18938#issuecomment-3696700131

   > πŸ‘‹ I think I found the issue, in short, we should check the strong count of the inner struct not the DynamicFilterPhysicalExpr πŸ€¦β€β™€οΈ since sometimes new DynamicFilterPhysicalExpr are created but still keep the same `inner` struct reference. I will push the updated commit in the same PR in a moment
   
   Ahhh yep that sounds right to me! E.g. `with_new_children` will create a new `Arc<DynamicFilterPhysicalExpr>` but the inner reference is what matters πŸ‘πŸ» 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org