You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2016/10/06 19:09:20 UTC
[jira] [Commented] (SPARK-17809) scala.MatchError: BooleanType when
casting a struct
[ https://issues.apache.org/jira/browse/SPARK-17809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15552864#comment-15552864 ]
Josh Rosen commented on SPARK-17809:
------------------------------------
For context, here's the stacktrace as of Spark 2.0.0:
{code}
scala.MatchError: BooleanType (of class org.apache.spark.sql.types.BooleanType$)
at org.apache.spark.sql.catalyst.expressions.Cast.castToBoolean(Cast.scala:148)
at org.apache.spark.sql.catalyst.expressions.Cast.org$apache$spark$sql$catalyst$expressions$Cast$$cast(Cast.scala:428)
at org.apache.spark.sql.catalyst.expressions.Cast$$anonfun$1.apply(Cast.scala:405)
at org.apache.spark.sql.catalyst.expressions.Cast$$anonfun$1.apply(Cast.scala:404)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
at org.apache.spark.sql.catalyst.expressions.Cast.castStruct(Cast.scala:404)
at org.apache.spark.sql.catalyst.expressions.Cast.org$apache$spark$sql$catalyst$expressions$Cast$$cast(Cast.scala:437)
at org.apache.spark.sql.catalyst.expressions.Cast.cast$lzycompute(Cast.scala:445)
at org.apache.spark.sql.catalyst.expressions.Cast.cast(Cast.scala:445)
at org.apache.spark.sql.catalyst.expressions.Cast.nullSafeEval(Cast.scala:447)
at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:324)
at org.apache.spark.sql.catalyst.expressions.Alias.eval(namedExpressions.scala:142)
at org.apache.spark.sql.catalyst.expressions.InterpretedProjection.apply(Projection.scala:45)
at org.apache.spark.sql.catalyst.expressions.InterpretedProjection.apply(Projection.scala:29)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.immutable.List.map(List.scala:285)
at org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation$$anonfun$apply$37.applyOrElse(Optimizer.scala:1484)
at org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation$$anonfun$apply$37.applyOrElse(Optimizer.scala:1480)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:279)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:279)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:278)
at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:268)
at org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation$.apply(Optimizer.scala:1480)
at org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation$.apply(Optimizer.scala:1479)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:85)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:82)
at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57)
at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66)
at scala.collection.mutable.WrappedArray.foldLeft(WrappedArray.scala:35)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:82)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:74)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:74)
at org.apache.spark.sql.execution.QueryExecution.optimizedPlan$lzycompute(QueryExecution.scala:74)
at org.apache.spark.sql.execution.QueryExecution.optimizedPlan(QueryExecution.scala:74)
at org.apache.spark.sql.execution.QueryExecution.sparkPlan$lzycompute(QueryExecution.scala:78)
at org.apache.spark.sql.execution.QueryExecution.sparkPlan(QueryExecution.scala:76)
at org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:83)
at org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:83)
at org.apache.spark.sql.Dataset.withCallback(Dataset.scala:2541)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collect(Dataset.scala:2187)
at org.apache.spark.sql.Dataset.collect(Dataset.scala:2163)
... 50 elided
{code}
> scala.MatchError: BooleanType when casting a struct
> ---------------------------------------------------
>
> Key: SPARK-17809
> URL: https://issues.apache.org/jira/browse/SPARK-17809
> Project: Spark
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: Niek Bartholomeus
>
> I have a Dataframe with a struct and I need to rename some fields to lower case before saving it to cassandra.
> It turns out that it's not possible to cast a boolean field of a struct to another boolean field in the renamed struct:
> {quote}
> case class ClassWithBoolean(flag: Boolean)
> case class Parent(cwb: ClassWithBoolean)
> val structCwb: DataType = StructType(Seq(
> StructField("flag", BooleanType, true)
> ))
> Seq(Parent(ClassWithBoolean(true)))
> .toDF
> .withColumn("cwb", $"cwb".cast(structCwb))
> .collect
> scala.MatchError: BooleanType (of class org.apache.spark.sql.types.BooleanType$)
> {quote}
> A workaround is to temporarily cast the field to an Integer and back:
> {quote}
> val structCwbTmp: DataType = StructType(Seq(
> StructField("flag", IntegerType, true)
> ))
> Seq(Parent(ClassWithBoolean(true)))
> .toDF
> .withColumn("cwb", $"cwb".cast(structCwbTmp))
> .withColumn("cwb", $"cwb".cast(structCwb))
> .collect
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org