[GitHub] [spark] gengliangwang opened a new pull request #24597: [SPARK-27698] Add new method for getting pushed down filters in Parquet file reader

[GitBox <gi...@apache.org>]

gengliangwang opened a new pull request #24597: [SPARK-27698] Add new method for getting pushed down filters in Parquet file reader
URL: https://github.com/apache/spark/pull/24597
 
 
   ## What changes were proposed in this pull request?
   
   To return accurate pushed filters in Parquet file scan(https://github.com/apache/spark/pull/24327#pullrequestreview-234775673), we can process the original data source filters in the following way:
   1. For "And" operators, split the conjunctive predicates and try converting each of them. After that
   1.1 if partially predicate pushed down is allowed, return convertible results; 
   1.2 otherwise, return the whole predicate if convertible, or empty result if not convertible.
   
   2. For other operators, they are not able to be partially pushed down. 
   2.1 if the entire predicate is convertible, return itself
   2.2 otherwise, return an empty result.
   
   ## How was this patch tested?
   
   Unit test
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org