[jira] [Created] (HUDI-8943) Partition stats should skip queries with null or complex expressions

["Lokesh Jain (Jira)" <ji...@apache.org>]

Lokesh Jain created HUDI-8943:
---------------------------------

             Summary: Partition stats should skip queries with null or complex expressions
                 Key: HUDI-8943
                 URL: https://issues.apache.org/jira/browse/HUDI-8943
             Project: Apache Hudi
          Issue Type: Bug
          Components: index
            Reporter: Lokesh Jain
            Assignee: Lokesh Jain


Currently partition stats avoids data skipping if even one of the query filters has a null, value count based expression. Or if any query uses a complex expression like expression index.
The jira aims to fix the logic so that index only filters out such queries and performs pruning based on the other queries.
For example Spark automatically adds query isNotNull on a column like:- isnotnull(driver#869) when an equality query on driver is used. We can ignore such queries for partition stats.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)