You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/03 20:42:20 UTC
[GitHub] [spark] huaxingao commented on pull request #33584: [SPARK-36351][SQL] Separate partition filters and data filters in PushDownUtils
huaxingao commented on pull request #33584:
URL: https://github.com/apache/spark/pull/33584#issuecomment-892151806
@gengliangwang
Thank you very much for taking a look!
I asked @cloud-fan offline, and he was suggesting to separate the partition filters and data filters in data source implementation. I guess what we can do is to separate partition filters and data filters at the time of push down filter, and only return the data filters as post scan filters.
```
override def pushFilters(filters: Array[Filter]): Array[Filter] = {
separate partition filters and data filters
this.filters = filters -- partition filters
this.filters
}
```
There is a problem, though. In the current implementation, `partitionFilters` and `dataFilters` in `FileScan` are `Expression`, and the partition pruning code takes `partitionFilters` in the format of `Expression`. If I separate the filters in `pushFilters`, the partition filters and data filters are in the format of `sources.Filter`. The data filters can be reconstructed back to `Expression` in `PushDownUtils`, but I don't have a `translatedFilterToExpr` map inside ScanBuilder to reconstruct partition filters to `Expression`. Seems to me I can't use the current partition pruning code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org