You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Max Kießling (JIRA)" <ji...@apache.org> on 2018/03/06 10:10:00 UTC

[jira] [Created] (SPARK-23610) Cast of ArrayType of NullType to ArrayType of nullable material type does not work

Max Kießling created SPARK-23610:
------------------------------------

             Summary: Cast of ArrayType of NullType to ArrayType of nullable material type does not work
                 Key: SPARK-23610
                 URL: https://issues.apache.org/jira/browse/SPARK-23610
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.2.0
            Reporter: Max Kießling


Given a DataFrame that contains a column with _ArrayType of NullType_
casting this column into ArrayType of any material nullable type (e.g. _ArrayType(LongType, true)_ ) should be possible.
{code}
it("can cast arrays of null type into arrays of nullable material types") {
 val inputData = Seq(
 Row(Array())
 ).asJava

val schema = StructType(Seq(
 StructField("list", ArrayType(NullType, true), false)
 ))

val data = caps.sparkSession.createDataFrame(inputData, schema)

data.withColumn("longList",data.col("list").cast(ArrayType(LongType, true))).show
 }
{code}

This test fails with the message: 

{noformat}
NullType (of class org.apache.spark.sql.types.NullType$)
 scala.MatchError: NullType (of class org.apache.spark.sql.types.NullType$)
 at org.apache.spark.sql.catalyst.expressions.Cast.castToLong(Cast.scala:310)
 at org.apache.spark.sql.catalyst.expressions.Cast.org$apache$spark$sql$catalyst$expressions$Cast$$cast(Cast.scala:516)
 at org.apache.spark.sql.catalyst.expressions.Cast.castArray(Cast.scala:455)
 at org.apache.spark.sql.catalyst.expressions.Cast.org$apache$spark$sql$catalyst$expressions$Cast$$cast(Cast.scala:519)
 at org.apache.spark.sql.catalyst.expressions.Cast.cast$lzycompute(Cast.scala:531)
 at org.apache.spark.sql.catalyst.expressions.Cast.cast(Cast.scala:531)
 at org.apache.spark.sql.catalyst.expressions.Cast.nullSafeEval(Cast.scala:533)
 at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:327)
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org