[jira] [Created] (SPARK-16689) FileSourceStrategy: Pruning Partition Columns When No Partition Column Exist in Project

["Xiao Li (JIRA)" <ji...@apache.org>]

Xiao Li created SPARK-16689:
-------------------------------

             Summary: FileSourceStrategy: Pruning Partition Columns When No Partition Column Exist in Project
                 Key: SPARK-16689
                 URL: https://issues.apache.org/jira/browse/SPARK-16689
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.0.0
            Reporter: Xiao Li


For partitioned file sources, the current implementation always scans all the partition columns. However, this is not necessary when the projected column list does not include any partition column. In addition, we also can avoid the unnecessary Project.

Below is an example,
{noformat}
spark
  .range(N)
  .selectExpr("id AS value1", "id AS value2", "id AS p1", "id AS p2", "id AS p3")
  .toDF("value", "value2", "p1", "p2", "p3").write.format("json")
  .partitionBy("p1", "p2", "p3").save(tempDir)

spark.read.format("json").load(tempDir).selectExpr("value")
{noformat}

Before the PR changes, the physical plan is like:
{noformat}
== Physical Plan ==
*Project [value#37L]
+- *Scan json [value#37L,p1#39,p2#40,p3#41] Format: JSON, InputPaths: file:/private/var/folders/4b/sgmfldk15js406vk7lw5llzw0000gn/T/spark-f7a4294a-2e1b-4f44-9ebb-1a5eb..., PushedFilters: [], ReadSchema: struct<value:bigint>
{noformat}

After the PR changes, the physical plan becomes:
{noformat}
== Physical Plan ==
*Scan json [value#147L] Format: JSON, InputPaths: file:/private/var/folders/4b/sgmfldk15js406vk7lw5llzw0000gn/T/spark-a5bcb14a-46c2-4c20-8f34-9662b..., PushedFilters: [], ReadSchema: struct<value:bigint>
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org