You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Max Kießling (JIRA)" <ji...@apache.org> on 2018/03/06 10:11:00 UTC
[jira] [Updated] (SPARK-23610) Cast of ArrayType of NullType to
ArrayType of nullable material type does not work
[ https://issues.apache.org/jira/browse/SPARK-23610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Max Kießling updated SPARK-23610:
---------------------------------
Description:
Given a DataFrame that contains a column with _ArrayType of NullType_
casting this column into ArrayType of any material nullable type (e.g. _ArrayType(LongType, true)_ ) should be possible.
{code}
it("can cast arrays of null type into arrays of nullable material types") {
val inputData = Seq(
Row(Array())
).asJava
val schema = StructType(Seq(
StructField("list", ArrayType(NullType, true), false)
))
val data = caps.sparkSession.createDataFrame(inputData, schema)
data.withColumn("longList",data.col("list").cast(ArrayType(LongType, true))).show
}
{code}
This test fails with the message:
{noformat}
NullType (of class org.apache.spark.sql.types.NullType$)
scala.MatchError: NullType (of class org.apache.spark.sql.types.NullType$)
at org.apache.spark.sql.catalyst.expressions.Cast.castToLong(Cast.scala:310)
at org.apache.spark.sql.catalyst.expressions.Cast.org$apache$spark$sql$catalyst$expressions$Cast$$cast(Cast.scala:516)
at org.apache.spark.sql.catalyst.expressions.Cast.castArray(Cast.scala:455)
at org.apache.spark.sql.catalyst.expressions.Cast.org$apache$spark$sql$catalyst$expressions$Cast$$cast(Cast.scala:519)
at org.apache.spark.sql.catalyst.expressions.Cast.cast$lzycompute(Cast.scala:531)
at org.apache.spark.sql.catalyst.expressions.Cast.cast(Cast.scala:531)
at org.apache.spark.sql.catalyst.expressions.Cast.nullSafeEval(Cast.scala:533)
at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:327)
{noformat}
was:
Given a DataFrame that contains a column with _ArrayType of NullType_
casting this column into ArrayType of any material nullable type (e.g. _ArrayType(LongType, true)_ ) should be possible.
{code}
it("can cast arrays of null type into arrays of nullable material types") {
val inputData = Seq(
Row(Array())
).asJava
val schema = StructType(Seq(
StructField("list", ArrayType(NullType, true), false)
))
val data = caps.sparkSession.createDataFrame(inputData, schema)
data.withColumn("longList",data.col("list").cast(ArrayType(LongType, true))).show
}
{code}
This test fails with the message:
{noformat}
NullType (of class org.apache.spark.sql.types.NullType$)
scala.MatchError: NullType (of class org.apache.spark.sql.types.NullType$)
at org.apache.spark.sql.catalyst.expressions.Cast.castToLong(Cast.scala:310)
at org.apache.spark.sql.catalyst.expressions.Cast.org$apache$spark$sql$catalyst$expressions$Cast$$cast(Cast.scala:516)
at org.apache.spark.sql.catalyst.expressions.Cast.castArray(Cast.scala:455)
at org.apache.spark.sql.catalyst.expressions.Cast.org$apache$spark$sql$catalyst$expressions$Cast$$cast(Cast.scala:519)
at org.apache.spark.sql.catalyst.expressions.Cast.cast$lzycompute(Cast.scala:531)
at org.apache.spark.sql.catalyst.expressions.Cast.cast(Cast.scala:531)
at org.apache.spark.sql.catalyst.expressions.Cast.nullSafeEval(Cast.scala:533)
at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:327)
{noformat}
> Cast of ArrayType of NullType to ArrayType of nullable material type does not work
> ----------------------------------------------------------------------------------
>
> Key: SPARK-23610
> URL: https://issues.apache.org/jira/browse/SPARK-23610
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Max Kießling
> Priority: Minor
>
> Given a DataFrame that contains a column with _ArrayType of NullType_
> casting this column into ArrayType of any material nullable type (e.g. _ArrayType(LongType, true)_ ) should be possible.
> {code}
> it("can cast arrays of null type into arrays of nullable material types") {
> val inputData = Seq(
> Row(Array())
> ).asJava
> val schema = StructType(Seq(
> StructField("list", ArrayType(NullType, true), false)
> ))
> val data = caps.sparkSession.createDataFrame(inputData, schema)
> data.withColumn("longList",data.col("list").cast(ArrayType(LongType, true))).show
> }
> {code}
> This test fails with the message:
> {noformat}
> NullType (of class org.apache.spark.sql.types.NullType$)
> scala.MatchError: NullType (of class org.apache.spark.sql.types.NullType$)
> at org.apache.spark.sql.catalyst.expressions.Cast.castToLong(Cast.scala:310)
> at org.apache.spark.sql.catalyst.expressions.Cast.org$apache$spark$sql$catalyst$expressions$Cast$$cast(Cast.scala:516)
> at org.apache.spark.sql.catalyst.expressions.Cast.castArray(Cast.scala:455)
> at org.apache.spark.sql.catalyst.expressions.Cast.org$apache$spark$sql$catalyst$expressions$Cast$$cast(Cast.scala:519)
> at org.apache.spark.sql.catalyst.expressions.Cast.cast$lzycompute(Cast.scala:531)
> at org.apache.spark.sql.catalyst.expressions.Cast.cast(Cast.scala:531)
> at org.apache.spark.sql.catalyst.expressions.Cast.nullSafeEval(Cast.scala:533)
> at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:327)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org