[jira] [Commented] (SPARK-14193) Skip unnecessary sorts if input data have been already ordered in InMemoryRelation

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-14193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214061#comment-15214061 ] 

Apache Spark commented on SPARK-14193:
--------------------------------------

User 'maropu' has created a pull request for this issue:
https://github.com/apache/spark/pull/11997

> Skip unnecessary sorts if input data have been already ordered in InMemoryRelation
> ----------------------------------------------------------------------------------
>
>                 Key: SPARK-14193
>                 URL: https://issues.apache.org/jira/browse/SPARK-14193
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 1.6.1
>            Reporter: Takeshi Yamamuro
>
> This ticket is to skip unnecessary sorts if input data have been already ordered in InMemoryTable.
> Let's say we have a cached table with column 'a' sorted;
> {code}
> val df1 = Seq((1, 0), (3, 0), (2, 0), (1, 0)).toDF("a", "b")
> val df2 = df1.sort("a").cache
> df2.show // just cache data
> {code}
> If you say `df2.sort("a")`, the current spark generates a plan like;
> {code}
> == Physical Plan ==
> Sort [a#13 ASC], true, 0
> +- InMemoryColumnarTableScan [a#13,b#14], InMemoryRelation [a#13,b#14], true, 10000, StorageLevel(true, true, false, true, 1), Sort [a#13 ASC], true, 0, None
> {code}
> This ticket targets at removing this unncessary sort.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org