You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Rastislav Krist (JIRA)" <ji...@apache.org> on 2016/04/01 14:09:25 UTC
[jira] [Comment Edited] (SPARK-8370) Add API for data sources to
register databases
[ https://issues.apache.org/jira/browse/SPARK-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221599#comment-15221599 ]
Rastislav Krist edited comment on SPARK-8370 at 4/1/16 12:09 PM:
-----------------------------------------------------------------
we would welcome this feature too (as we are using non-standard data sources). For now, we have implemented a workaround which relies in mixing a custom trait into SQLContext/HiveContext on creation:
{code:title=CustomHierarchyCatalogSupport.scala|borderStyle=solid}
package org.apache.spark.sql
import org.apache.spark.sql.catalyst.analysis.{Catalog, CustomHierarchyCatalog, SimpleCatalog}
trait CustomHierarchyCatalogSupport { self: SQLContext =>
@transient
override lazy val catalog: Catalog with CustomHierarchyCatalog = new SimpleCatalog(conf) with CustomHierarchyCatalog
def attachDatabase(position: String, tables: Catalog): Unit =
catalog.asInstanceOf[CustomHierarchyCatalog].registerChildCatalog(position, tables)
}
{code}
where CustomHierarchyCatalog is a special Catalog implementation supporting building of hierarchical tree of Catalogs. I can provide the full implementation, if desired (2 classes in total) - works like a charm but gets occasionally broken when Spark SQL internal API changes on version upgrades.
was (Author: rkrist):
we would welcome this feature too (as we are using non-standard data sources). For now, we have implemented a workaround which relies in mixing a custom trait into SQLContext/HiveContext on creation:
package org.apache.spark.sql
import org.apache.spark.sql.catalyst.analysis.{Catalog, CustomHierarchyCatalog, SimpleCatalog}
trait CustomHierarchyCatalogSupport { self: SQLContext =>
@transient
override lazy val catalog: Catalog with CustomHierarchyCatalog = new SimpleCatalog(conf) with CustomHierarchyCatalog
def attachDatabase(position: String, tables: Catalog): Unit =
catalog.asInstanceOf[CustomHierarchyCatalog].registerChildCatalog(position, tables)
}
where CustomHierarchyCatalog is a special Catalog implementation supporting building of hierarchical tree of Catalogs. I can provide the full implementation, if desired (2 classes in total) - works like a charm but gets occasionally broken when Spark SQL internal API changes on version upgrades.
> Add API for data sources to register databases
> ----------------------------------------------
>
> Key: SPARK-8370
> URL: https://issues.apache.org/jira/browse/SPARK-8370
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Reporter: Santiago M. Mola
>
> This API would allow to register a database with a data source instead of just a table. Registering a data source database would register all its table and maintain the catalog updated. The catalog could delegate to the data source lookups of tables for a database registered with this API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org