You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xiao Li (JIRA)" <ji...@apache.org> on 2015/10/13 23:32:05 UTC

[jira] [Commented] (SPARK-10217) Spark SQL cannot handle ordering directive in ORDER BY clauses with expressions

    [ https://issues.apache.org/jira/browse/SPARK-10217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955723#comment-14955723 ] 

Xiao Li commented on SPARK-10217:
---------------------------------

Hi, Simeon, 

I am trying to reproduce your problem on Spark 1.5.1. I failed to get the exception you got. See the plan I got:

== Parsed Logical Plan ==
'Sort [('c1 + 'c1) ASC], true
 'Project [unresolvedalias('c1)]
  'UnresolvedRelation [inMemoryDF], None

== Analyzed Logical Plan ==
c1: string
Sort [(cast(c1#0 as double) + cast(c1#0 as double)) ASC], true
 Project [c1#0]
  Subquery inMemoryDF
   LogicalRDD [c1#0,c2#1,c3#2,c4#3], MapPartitionsRDD[1] at createDataFrame at SimpleApp.scala:42

== Optimized Logical Plan ==
Sort [(cast(c1#0 as double) + cast(c1#0 as double)) ASC], true
 Project [c1#0]
  LogicalRDD [c1#0,c2#1,c3#2,c4#3], MapPartitionsRDD[1] at createDataFrame at SimpleApp.scala:42

== Physical Plan ==
TungstenSort [(cast(c1#0 as double) + cast(c1#0 as double)) ASC], true, 0
 ConvertToUnsafe
  Exchange rangepartitioning((cast(c1#0 as double) + cast(c1#0 as double)) ASC)
   ConvertToSafe
    TungstenProject [c1#0]
     Scan PhysicalRDD[c1#0,c2#1,c3#2,c4#3]

Thanks, 

Xiao Li

> Spark SQL cannot handle ordering directive in ORDER BY clauses with expressions
> -------------------------------------------------------------------------------
>
>                 Key: SPARK-10217
>                 URL: https://issues.apache.org/jira/browse/SPARK-10217
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.4.1
>         Environment: Ubuntu on AWS
>            Reporter: Simeon Simeonov
>              Labels: SQL, analyzers
>
> Spark SQL supports expressions in ORDER BY clauses, e.g.,
> {code}
> scala> sqlContext.sql("select cnt from test order by (cnt + cnt)")
> res2: org.apache.spark.sql.DataFrame = [cnt: bigint]
> {code}
> However, the analyzer gets confused when there is an explicit ordering directive (ASC/DESC):
> {code}
> scala> sqlContext.sql("select cnt from test order by (cnt + cnt) asc")
> 15/08/25 04:08:02 INFO ParseDriver: Parsing command: select cnt from test order by (cnt + cnt) asc
> org.apache.spark.sql.AnalysisException: extraneous input 'asc' expecting EOF near '<EOF>'; line 1 pos 40
> 	at org.apache.spark.sql.hive.HiveQl$.createPlan(HiveQl.scala:289)
> 	at org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:41)
> 	at org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:40)
>         ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org