[jira] [Commented] (SPARK-13929) Use Scala reflection for UDFs

["Joan Goyeau (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-13929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221656#comment-15221656 ] 

Joan Goyeau commented on SPARK-13929:
-------------------------------------

I investigated a way to implement Scala reflection for the part you show which involves Annotations.

This is a bit tricky because reflection of annotations in Scala is for now only to the AST level.
What we can do is mix both Scala and Java reflection which does something like that:
```scala
case t if getClassFromType(t).isAnnotationPresent(classOf[SQLUserDefinedType]) =>
  val udt = getClassFromType(t).getAnnotation(classOf[SQLUserDefinedType]).udt().newInstance()
```
But then `getClassFromType` is using `runtimeClass` and therefore we need to move `schemaFor` from the `ScalaReflection` trait (compile time and runtime universe) to his companion object (runtime universe) as it works only in the runtime universe.

Let me know your thoughts.

> Use Scala reflection for UDFs
> -----------------------------
>
>                 Key: SPARK-13929
>                 URL: https://issues.apache.org/jira/browse/SPARK-13929
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: Jakob Odersky
>            Priority: Minor
>
> {{ScalaReflection}} uses native Java reflection for User Defined Types which would fail if such types are not plain Scala classes that map 1:1 to Java.
> Consider the following extract (from here https://github.com/apache/spark/blob/92024797a4fad594b5314f3f3be5c6be2434de8a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala#L376 ):
> {code}
> case t if Utils.classIsLoadable(className) &&
> Utils.classForName(className).isAnnotationPresent(classOf[SQLUserDefinedType]) =>
> val udt = Utils.classForName(className).getAnnotation(classOf[SQLUserDefinedType]).udt().newInstance()
> //...
> {code}
> If {{t}}'s runtime class is actually synthetic (something that doesn't exist in Java and hence uses a dollar sign internally), such as nested classes or package objects, the above code will fail.
> Currently there are no known use-cases of synthetic user-defined types (hence the minor priority), however it would be best practice to remove plain Java reflection and rely on Scala reflection instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org