[GitHub] [spark] LorenzoMartini commented on pull request #32921: [SPARK-35779][SQL] Dynamic filtering for Data Source V2

["LorenzoMartini (via GitHub)" <gi...@apache.org>]

LorenzoMartini commented on PR #32921:
URL: https://github.com/apache/spark/pull/32921#issuecomment-1418837654

   Hey @aokolnychyi thank you for the answer. I see that those sources have special optimizations. However we do have instances of data transformations being incredibly slower using spark's v2 datasources and the only difference in the query plans compared to those same transformation ran using v1 datasources is the absence of the dynamic pruning expressions. Do you have any suggestions on how to improve those use cases if not trying to implement `SupportsRuntimeFiltering` for the spark sources? Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org