[jira] [Commented] (SPARK-18457) ORC and other columnar formats using HiveShim read all columns when doing a simple count

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-18457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669150#comment-15669150 ] 

Hyukjin Kwon commented on SPARK-18457:
--------------------------------------

Hm, don't we push down no column when performing `count()`? 

{code}
val data = (0 to 255).zip(0 to 255).toDF("a", "b")
val path = "/tmp/aa"
data.write.orc(path)
spark.read.orc(path).createOrReplaceTempView("a_orc_table")
spark.sql("select count(*) from a_orc_table").explain(true)
{code}

{code}
== Parsed Logical Plan ==
'Project [unresolvedalias('count(1), None)]
+- 'UnresolvedRelation `a_orc_table`

== Analyzed Logical Plan ==
count(1): bigint
Aggregate [count(1) AS count(1)#28L]
+- SubqueryAlias a_orc_table
   +- Relation[a#16,b#17] orc

== Optimized Logical Plan ==
Aggregate [count(1) AS count(1)#28L]
+- Project
   +- Relation[a#16,b#17] orc

== Physical Plan ==
*HashAggregate(keys=[], functions=[count(1)], output=[count(1)#28L])
+- Exchange SinglePartition
   +- *HashAggregate(keys=[], functions=[partial_count(1)], output=[count#30L])
      +- *FileScan orc [] Batched: false, Format: ORC, Location: InMemoryFileIndex[file:/tmp/aa], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<>
{code}

It seems ORC datasource does not try to read all columns and I just verified this by debugging with IDE.

> ORC and other columnar formats using HiveShim read all columns when doing a simple count
> ----------------------------------------------------------------------------------------
>
>                 Key: SPARK-18457
>                 URL: https://issues.apache.org/jira/browse/SPARK-18457
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.3, 2.0.2
>         Environment: Hadoop 2.7.0
>            Reporter: Andrew Ray
>            Priority: Minor
>
> Doing a `select count(*) from a_orc_table` reads all columns and thus is slower than a query selecting one like `select count(a_column) from a_orc_table`. Data read can be seen in the UI (appears to only be accurate for Hadoop 2.5+ based on comment in FileScanRDD.scala line 80).
> I will create a PR shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org