You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Katherine Prevost (JIRA)" <ji...@apache.org> on 2015/10/30 20:58:27 UTC
[jira] [Commented] (SPARK-11010) Fixes and enhancements addressing
UDTs' api and several usability concerns
[ https://issues.apache.org/jira/browse/SPARK-11010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983192#comment-14983192 ]
Katherine Prevost commented on SPARK-11010:
-------------------------------------------
In addition to the difficulties with UDTs meant to support 3rd-party types, there's also currently a problem if you wish to provide UDTs on home-grown types without requiring Spark as a dependency for that type. Annotating a class with SQLUserDefinedType causes that class to require SQLUserDefinedType to be available at load time. If the underlying class doesn't know or care about Spark, this is rather painful and not at all what you want. You can still use the UDT without the annotation—but it requires all use of the UDT to be done via explicitly defining your schemas.
Any solution that solves the 3rd party or extensions problem will probably also solve the load-time dependency problem.
> Fixes and enhancements addressing UDTs' api and several usability concerns
> --------------------------------------------------------------------------
>
> Key: SPARK-11010
> URL: https://issues.apache.org/jira/browse/SPARK-11010
> Project: Spark
> Issue Type: Epic
> Components: SQL
> Affects Versions: 1.5.0, 1.5.1
> Reporter: John Muller
> Labels: UDT
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Catalyst SQL types allow for UserDefinedTypes, but do not allow for easy extension of 3rd party types or extensions to built-in types like DecimalType or StringType (private classes).
> Additionally, the API can infer much more of what's needed from the type parameter than it currently does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org