You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Adrian Wang (JIRA)" <ji...@apache.org> on 2015/10/12 09:35:05 UTC
[jira] [Commented] (SPARK-10507) reject temporal expressions such
as timestamp - timestamp at parse time
[ https://issues.apache.org/jira/browse/SPARK-10507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952747#comment-14952747 ]
Adrian Wang commented on SPARK-10507:
-------------------------------------
It seems this bug has been fixed long long ago. I just checked 1.5.0 and there's no such problem.
> reject temporal expressions such as timestamp - timestamp at parse time
> ------------------------------------------------------------------------
>
> Key: SPARK-10507
> URL: https://issues.apache.org/jira/browse/SPARK-10507
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.3.1
> Reporter: N Campbell
> Priority: Minor
>
> TIMESTAMP - TIMESTAMP in ISO-SQL should return an interval type which SPARK does not support..
> A similar expression in Hive 0.13 fails with Error: Could not create ResultSet: Required field 'type' is unset! Struct:TPrimitiveTypeEntry(type:null) and SPARK has similar "challenges". While Hive 1.2.1 has added some interval type support it is far from complete with respect to ISO-SQL.
> The ability to compute the period of time (years, days, weeks, hours, ...) between timestamps or add/substract intervals from a timestamp are extremely common in business applications.
> Currently, a value expression such as select timestampcol - timestampcol from t will fail during execution and not parse time. While the error thrown states that fact, it would better for those value expressions to be rejected at parse time along with indicating the expression that is causing the parser error.
> Operation: execute
> Errors:
> {code}
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6214.0 failed 4 times, most recent failure: Lost task 0.3 in stage 6214.0 (TID 21208, sandbox.hortonworks.com): java.lang.RuntimeException: Type TimestampType does not support numeric operations
> at scala.sys.package$.error(package.scala:27)
> at org.apache.spark.sql.catalyst.expressions.Subtract.numeric$lzycompute(arithmetic.scala:138)
> at org.apache.spark.sql.catalyst.expressions.Subtract.numeric(arithmetic.scala:136)
> at org.apache.spark.sql.catalyst.expressions.Subtract.eval(arithmetic.scala:150)
> at org.apache.spark.sql.catalyst.expressions.Alias.eval(namedExpressions.scala:113)
> at org.apache.spark.sql.catalyst.expressions.InterpretedMutableProjection.apply(Projection.scala:68)
> at org.apache.spark.sql.catalyst.expressions.InterpretedMutableProjection.apply(Projection.scala:52)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
> at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
> at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
> at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
> at scala.collection.AbstractIterator.to(Iterator.scala:1157)
> at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
> at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
> at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
> at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
> at org.apache.spark.rdd.RDD$$anonfun$17.apply(RDD.scala:813)
> at org.apache.spark.rdd.RDD$$anonfun$17.apply(RDD.scala:813)
> at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1498)
> {code}
> {code}
> create table if not exists TTS ( RNUM int , CTS timestamp )TERMINATED BY '\n'
> STORED AS orc ;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org