[I] Consider add already pushed filters to those that passed to `supports_filters_pushdown` [datafusion]

["askalt (via GitHub)" <gi...@apache.org>]

askalt opened a new issue, #16075:
URL: https://github.com/apache/datafusion/issues/16075

   
   Consider the next use-case: there is table t(a, b) and some index built for this table. The index can support exactly (`TableProviderFilterPushDown::Exact`) only a single predicate. For example, it can support `a == 1` or `b > 1` but not both at the same time (it is the index itself nature). 
   
   For now, `push_down_filter` do not account that some filters were pushed down earlier. For example, if we implement the use-case above as a table provider, which does:
   
   ```Rust
   fn supports_filters_pushdown(
       &self,
       filters: &[&Expr],
   ) -> Result<Vec<TableProviderFilterPushDown>> {
       // Can support only single filter.
       Ok(if filters.is_empty() {
           vec![]
       } else {
           let mut push = vec![TableProviderFilterPushDown::Exact; 1];
           for i in 0..filters.len() - 1 {
               push.push(TableProviderFilterPushDown::Unsupported);
           }
           push
       })
   }
   ```
   
   And there is a query `SELECT * FROM t WHERE a > 1 AND b == 1` -- it will return [`Exact`, `Unsupported`] for the first `supports_filters_pushdown` call, and then, if the optimizer will try to push down filters again, it will pass `b == 1` and got `Exact`. But it is not true for the bunch of filters.
   
   I thought about workarounds, for example:
   
   - Remember which filters were pushed in the table provider itself. It is not working for the scenario when there are multiple parallel queries in this session.
   - Return `Inexact` always for such filters. It does not satisfy, because some filters are really could be supported.
   
   I suggest to pass already pushed filters into the `supports_filters_pushdown` and rebuild the list from the scratch each time. Please, highlight the pitfalls or tell that it is ok and I will work on the patch.
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org