[GitHub] [spark] cloud-fan commented on issue #19424: [SPARK-22197][SQL] push down operators to data source before planning

[GitBox <gi...@apache.org>]

cloud-fan commented on issue #19424: [SPARK-22197][SQL] push down operators to data source before planning
URL: https://github.com/apache/spark/pull/19424#issuecomment-540890347
 
 
   Usually a data source scans its data incrementally. So when the query has a limit, Spark stops consuming the iterator from data source, and data source won't scan all the data.
   
   But limit pushdown does have use cases. For now the only way is to write a catalyst rule, to catch the limit + scan query plan, and convert it to a custom query plan with limit pushed down.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org