[jira] [Resolved] (SPARK-16596) Refactor DataSourceScanExec to do partition discovery at execution instead of planning time

["Davies Liu (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-16596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Davies Liu resolved SPARK-16596.
--------------------------------
       Resolution: Fixed
    Fix Version/s: 2.1.0

Issue resolved by pull request 14241
[https://github.com/apache/spark/pull/14241]

> Refactor DataSourceScanExec to do partition discovery at execution instead of planning time
> -------------------------------------------------------------------------------------------
>
>                 Key: SPARK-16596
>                 URL: https://issues.apache.org/jira/browse/SPARK-16596
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Eric Liang
>            Priority: Minor
>             Fix For: 2.1.0
>
>
> Partition discovery is rather expensive, so we should do it at execution time instead of during physical planning. Right now there is not much benefit since ListingFileCatalog will read scan for all partitions at planning time anyways, but this can be optimized in the future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org