You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Eric Liang (JIRA)" <ji...@apache.org> on 2016/11/30 23:43:58 UTC

[jira] [Created] (SPARK-18661) Creating a partitioned datasource table should not scan all files in filesystem

Eric Liang created SPARK-18661:
----------------------------------

             Summary: Creating a partitioned datasource table should not scan all files in filesystem
                 Key: SPARK-18661
                 URL: https://issues.apache.org/jira/browse/SPARK-18661
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.1.0
            Reporter: Eric Liang
            Priority: Blocker


Even though in 2.1 creating a partitioned datasource table will not populate the partition data by default (until the user issues MSCK REPAIR TABLE), it seems we still scan the filesystem for no good reason.

We should avoid doing this when the user specifies a schema.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org