kotlin-spark-api

[Pasha Finkelshteyn <pa...@gmail.com>]

Hi everybody!

Me again. I'm developing Spark API for Kotlin and found interesting
behaviour:

As you know, Kotlin has concept of nullable types which I'm using in my
implementation of serializers/deserializers (slightly changed
ScalaReflection to be more concrete). But in one case it doesn't work
for me: when list contains nullable type

It outputs error like this:

Caused by: java.lang.NullPointerException: Null value appeared in
non-nullable field:
top level Product or row object
If the schema is inferred from a Scala tuple/case class, or a Java bean,
please try to use scala.Option[_] or other nullable types (e.g.
java.lang.Integer instead of int/scala.Int).

What is even more interesting that if I disable
"spark.sql.codegen.wholeStage" than program works as intended.

Any ideas on this?

Thanks,
Pasha

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org