You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "koert kuipers (JIRA)" <ji...@apache.org> on 2016/03/24 23:39:25 UTC
[jira] [Created] (SPARK-14139) Dataset loses nullability in
operations with RowEncoder
koert kuipers created SPARK-14139:
-------------------------------------
Summary: Dataset loses nullability in operations with RowEncoder
Key: SPARK-14139
URL: https://issues.apache.org/jira/browse/SPARK-14139
Project: Spark
Issue Type: Bug
Components: SQL
Reporter: koert kuipers
Priority: Minor
When i do
{noformat}
val df1 = sc.makeRDD(1 to 3).toDF
val df2 = df1.map(row => Row(row(0).asInstanceOf[Int] + 1))(RowEncoder(df1.schema))
println(s"schema before ${df1.schema} and after ${df2.schema}")
{noformat}
I get:
{noformat}
schema before StructType(StructField(value,IntegerType,false)) and after StructType(StructField(value,IntegerType,true))
{noformat}
The change in field nullable is unexpected and i consider it a bug.
This bug was introduced in:
[SPARK-13244][SQL] Migrates DataFrame to Dataset
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org