You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2016/10/03 23:22:20 UTC
[jira] [Comment Edited] (SPARK-17767) Spark SQL ExternalCatalog API
custom implementation support
[ https://issues.apache.org/jira/browse/SPARK-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15543733#comment-15543733 ]
Reynold Xin edited comment on SPARK-17767 at 10/3/16 11:22 PM:
---------------------------------------------------------------
Guys - while I think this is very useful to do, I'm going to mark this as "later" for now. The reason is that there are a lot of things to consider before making this switch, including:
- The ExternalCatalog API is currently internal, and we can't just make it public without thinking about the consequences and whether this API is maintainable in the long run.
- SPARK-15777 We need to design this in the context of catalog federation and persistence.
- SPARK-15691 Refactoring of how we integrate with Hive.
This is not as simple as just submitting a PR to make it pluggable.
was (Author: rxin):
Guys - while I think this is very useful to do, I'm going to mark this as "later" for now. The reason is that there are a lot of things to consider before making this switch, including:
- The ExternalCatalog API is currently internal, and we can't just make it public without thinking about the consequences.
- SPARK-15777 We need to design this in the context of catalog federation and persistence.
- SPARK-15691 Refactoring of how we integrate with Hive.
> Spark SQL ExternalCatalog API custom implementation support
> -----------------------------------------------------------
>
> Key: SPARK-17767
> URL: https://issues.apache.org/jira/browse/SPARK-17767
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Alex Liu
>
> There is no way/easy way to configure Spark to use customized ExternalCatalog. Internal source code is hardcoded to use either hive or in-memory metastore. Spark SQL thriftserver is hardcoded to use HiveExternalCatalog. We should be able to create a custom external catalog and thriftserver should be able to use it. Potentially Spark SQL thriftserver shouldn't depend on Hive thriftserer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org