You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by maropu <gi...@git.apache.org> on 2018/10/04 03:43:57 UTC

[GitHub] spark pull request #22512: [SPARK-25498][SQL][WIP] Fix SQLQueryTestSuite fai...

Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22512#discussion_r222530205
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ---
    @@ -238,7 +262,9 @@ object DecimalLiteral {
     /**
      * In order to do type checking, use Literal.create() instead of constructor
      */
    -case class Literal (value: Any, dataType: DataType) extends LeafExpression {
    +case class Literal(value: Any, dataType: DataType) extends LeafExpression {
    +
    +  Literal.validateLiteralValue(value, dataType)
    --- End diff --
    
    I'm not sure though, is it ok for `Literal` to have a different Scala-typed value for the corresponding `dataType`? , e.g., `new Literal(1 /* int value */, LongType)`? In the current master, there are some places to do so, e.g., 
    https://github.com/apache/spark/blob/927e527934a882fab89ca661c4eb31f84c45d830/sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeColumnCommand.scala#L213
    https://github.com/apache/spark/blob/927e527934a882fab89ca661c4eb31f84c45d830/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala#L796
    
    In the codegen path, this is ok because we add a correct literal suffix in `Literal.doGenCode` (e.g., `1L` for `new Literal(1, LongType)`);
    https://github.com/apache/spark/blob/927e527934a882fab89ca661c4eb31f84c45d830/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala#L294
    
    But, in the non-codegen path (e.g., `spark.sql.codegen.factoryMode=NO_CODEGEN` and `ConstantFolding`), this case throws an exception ;
    ```
    
    scala> import org.apache.spark.sql.Column
    scala> import org.apache.spark.sql.catalyst.expressions.Literal
    scala> import org.apache.spark.sql.types._
    scala> val intOne: Int = 1
    scala> val lit = Literal.create(intOne, LongType)
    scala> spark.range(1).select(struct(new Column(lit))).collect
    18/10/04 11:35:56 ERROR Executor: Exception in task 3.0 in stage 0.0 (TID 3)
    java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
    	at scala.runtime.BoxesRunTime.unboxToLong(BoxesRunTime.java:105)
    	at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow$class.getLong(rows.scala:42)
    	at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getLong(rows.scala:195)
    	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
    	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$11$$anon$1.hasNext(WholeStageCodegenExec.scala:619)
        ...
    ```
    
    WDYT? cc: @gatorsmile @cloud-fan


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org