You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2015/09/24 12:27:04 UTC

[jira] [Commented] (SPARK-10793) Make sparks use/subclassing of hive more maintainable

    [ https://issues.apache.org/jira/browse/SPARK-10793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906172#comment-14906172 ] 

Steve Loughran commented on SPARK-10793:
----------------------------------------

Leaving the sqp/hive parser integration alone, this is what can be done by making some basic changes upstream

# move spark up the same version of kryo that hive uses, currently 2.22. This has to be done via an upgrade to chill: [https://github.com/steveloughran/chill/tree/feature/support-kryo-2.22]
# hive thrift server services to call protected methods to create various classes (e.g. the cli), calls which can be overridden in subclasses: [https://github.com/steveloughran/hive/tree/stevel/1.2.2-SNAPSHOT-managed-thriftserver]
# spark code to directly import hive artifacts, and the hive thriftserver subclasses to override the creator methods to build up their service without any reflection. [https://github.com/steveloughran/spark/tree/stevel/feature/SPARK-10793-managed-hive]

> Make sparks use/subclassing of hive more maintainable
> -----------------------------------------------------
>
>                 Key: SPARK-10793
>                 URL: https://issues.apache.org/jira/browse/SPARK-10793
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 1.5.0
>            Reporter: Steve Loughran
>
> The latest spark/hive integration round has closed the gap with Hive versions, but the integration is still pretty complex
> # SparkSQL has deep hooks into the parser
> # hivethriftserver uses "aggressive reflection" to inject spark classes into the Hive base classes.
> # there's a separate org.sparkproject.hive JAR to isolate Kryo versions while avoiding the hive uberjar with all its dependencies getting into the spark uberjar.
> We can improve this with some assistance from the other projects, even though no guarantees of stability of things like the parser and thrift server APIs are likely in the near future



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org