[GitHub] [spark] peter-toth commented on pull request #38640: [SPARK-41124][SQL][TEST] Add DSv2 PlanStabilitySuites

[GitBox <gi...@apache.org>]

peter-toth commented on PR #38640:
URL: https://github.com/apache/spark/pull/38640#issuecomment-1312731989

   Side note: There seem to be many issues with V2 Scan construction in batch `Early Filter and Projection Push-Down` currently. As that is the only place where the scans are constructed the batch seems to be:
   - too late for `PushDownLeftSemiAntiJoin` (see the issue above with some of the queries)
   - and too early for `RewritePredicateSubquery` + `ColumnPruning` in batch `RewriteSubquery` as obviuosly the scans should be (re)constructed after the latest `ColumnPruning` or filter modifying rule to not return unecessary data. (This actually causes serious performance degradation in q94, q16 with DSv2 sources.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org