You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by marmbrus <gi...@git.apache.org> on 2014/12/20 22:02:44 UTC
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
GitHub user marmbrus opened a pull request:
https://github.com/apache/spark/pull/3752
[SPARK-4912][SQL] Persistent tables for the Spark SQL data sources
This is the first of a few improvements I'd like to make to the data sources API. I'll note that this does not correctly handle multiple databases. However, since much of the code does not, and since I don't think improving this will change the public API, I think its reasonable to handle that in a separate PR.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/marmbrus/spark persistentDataSourceTables
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3752.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3752
----
commit 677e8b37b87903f4770f7a7429e1e125aab27c12
Author: Michael Armbrust <mi...@databricks.com>
Date: 2014-12-20T20:45:28Z
First draft of persistent tables.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3752#discussion_r22159720
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -55,8 +56,60 @@ private[hive] class HiveMetastoreCatalog(hive: HiveContext) extends Catalog with
/** Connection to hive metastore. Usages should lock on `this`. */
protected[hive] val client = Hive.get(hive.hiveconf)
+ // TODO: Use this everywhere instead of tuples or databaseName, tableName,.
+ /** A fully qualified identifier for a table (i.e., database.tableName) */
+ case class TableIdent(database: String, name: String) {
+ def toLowerCase = TableIdent(database.toLowerCase, name.toLowerCase)
+ }
+
+ /** A cache of Spark SQL data source tables that have been accessed. */
+ protected[hive] val cachedDataSourceTables = CacheBuilder.newBuilder()
+ .maximumSize(1000)
+ .build(
+ new CacheLoader[TableIdent, LogicalPlan]() {
+ override def load(in: TableIdent): LogicalPlan = {
+ logDebug(s"Creating new cached data source for $in")
+ val table = client.getTable(in.database, in.name)
+
+ // It does not appear that the ql client for the metastore has a way to enumerate all the
+ // SerDe properties directly...
+ val method = classOf[Table].getDeclaredMethod("getSerdeInfo")
--- End diff --
This can be used to retrieve all SerDe properties:
```scala
table.getTTable.getSd.getSerdeInfo.getParameters
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3752#issuecomment-67884749
[Test build #24708 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24708/consoleFull) for PR 3752 at commit [`563ff40`](https://github.com/apache/spark/commit/563ff401c63c7345baf141dd19851d3dd7c9af08).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class TableIdent(database: String, name: String) `
* `case class CreateMetastoreDataSource(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3752#discussion_r22204044
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -55,8 +56,60 @@ private[hive] class HiveMetastoreCatalog(hive: HiveContext) extends Catalog with
/** Connection to hive metastore. Usages should lock on `this`. */
protected[hive] val client = Hive.get(hive.hiveconf)
+ // TODO: Use this everywhere instead of tuples or databaseName, tableName,.
+ /** A fully qualified identifier for a table (i.e., database.tableName) */
+ case class TableIdent(database: String, name: String) {
+ def toLowerCase = TableIdent(database.toLowerCase, name.toLowerCase)
+ }
+
+ /** A cache of Spark SQL data source tables that have been accessed. */
+ protected[hive] val cachedDataSourceTables = CacheBuilder.newBuilder()
+ .maximumSize(1000)
+ .build(
+ new CacheLoader[TableIdent, LogicalPlan]() {
+ override def load(in: TableIdent): LogicalPlan = {
+ logDebug(s"Creating new cached data source for $in")
+ val table = client.getTable(in.database, in.name)
+
+ // It does not appear that the ql client for the metastore has a way to enumerate all the
+ // SerDe properties directly...
+ val method = classOf[Table].getDeclaredMethod("getSerdeInfo")
--- End diff --
Thanks for the explanations, then this sounds good to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3752#issuecomment-67750989
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24677/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3752#issuecomment-67876256
[Test build #24707 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24707/consoleFull) for PR 3752 at commit [`e23f8fb`](https://github.com/apache/spark/commit/e23f8fbd039348e4d2f3c60335a3d60f9c61e19d).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3752#issuecomment-67787708
[Test build #24689 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24689/consoleFull) for PR 3752 at commit [`1002d20`](https://github.com/apache/spark/commit/1002d20e1648e84a0a5e812d87190ac1439cb702).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3752#issuecomment-67749769
[Test build #24677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24677/consoleFull) for PR 3752 at commit [`055869e`](https://github.com/apache/spark/commit/055869e02d77c1d0f18bc9671e02234e38b26cc2).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus closed the pull request at:
https://github.com/apache/spark/pull/3752
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3752#discussion_r22160548
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -55,8 +56,60 @@ private[hive] class HiveMetastoreCatalog(hive: HiveContext) extends Catalog with
/** Connection to hive metastore. Usages should lock on `this`. */
protected[hive] val client = Hive.get(hive.hiveconf)
+ // TODO: Use this everywhere instead of tuples or databaseName, tableName,.
+ /** A fully qualified identifier for a table (i.e., database.tableName) */
+ case class TableIdent(database: String, name: String) {
+ def toLowerCase = TableIdent(database.toLowerCase, name.toLowerCase)
+ }
+
+ /** A cache of Spark SQL data source tables that have been accessed. */
+ protected[hive] val cachedDataSourceTables = CacheBuilder.newBuilder()
+ .maximumSize(1000)
+ .build(
+ new CacheLoader[TableIdent, LogicalPlan]() {
+ override def load(in: TableIdent): LogicalPlan = {
+ logDebug(s"Creating new cached data source for $in")
+ val table = client.getTable(in.database, in.name)
+
+ // It does not appear that the ql client for the metastore has a way to enumerate all the
+ // SerDe properties directly...
+ val method = classOf[Table].getDeclaredMethod("getSerdeInfo")
+ method.setAccessible(true)
+ val serdeInfo = method.invoke(table).asInstanceOf[SerDeInfo]
+
+ val resolvedRelation =
+ ResolvedDataSource(
+ hive,
+ table.getProperty("spark.sql.sources.provider"),
+ serdeInfo.getParameters.toMap)
+
+ LogicalRelation(resolvedRelation.relation)
+ }
+ })
+
+ def refreshTable(databaseName: String, tableName: String): Unit = {
+ cachedDataSourceTables.refresh(TableIdent(databaseName, tableName).toLowerCase)
+ }
+
+ def invalidateTable(databaseName: String, tableName: String): Unit = {
+ cachedDataSourceTables.invalidate(TableIdent(databaseName, tableName).toLowerCase)
+ }
+
val caseSensitive: Boolean = false
+ def createDataSourceTable(tableName: String, provider: String, options: Map[String, String]) = {
+ val (dbName, tblName) = processDatabaseAndTableName("default", tableName)
+ val tbl = new Table(dbName, tblName)
+
+ tbl.setProperty("spark.sql.sources.provider", provider)
+ options.foreach { case (key, value) => tbl.setSerdeParam(key, value) }
+
+ // create the table
+ synchronized {
+ client.createTable(tbl, false)
--- End diff --
Should we mark tables from external data sources as external table? We can do this by
```scala
tbl.putToParameters("EXTERNAL", "TRUE")
tbl.setTableType(TableType.EXTERNAL_TABLE.toString())
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3752#discussion_r22183673
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -55,8 +56,60 @@ private[hive] class HiveMetastoreCatalog(hive: HiveContext) extends Catalog with
/** Connection to hive metastore. Usages should lock on `this`. */
protected[hive] val client = Hive.get(hive.hiveconf)
+ // TODO: Use this everywhere instead of tuples or databaseName, tableName,.
+ /** A fully qualified identifier for a table (i.e., database.tableName) */
+ case class TableIdent(database: String, name: String) {
+ def toLowerCase = TableIdent(database.toLowerCase, name.toLowerCase)
+ }
+
+ /** A cache of Spark SQL data source tables that have been accessed. */
+ protected[hive] val cachedDataSourceTables = CacheBuilder.newBuilder()
+ .maximumSize(1000)
+ .build(
+ new CacheLoader[TableIdent, LogicalPlan]() {
+ override def load(in: TableIdent): LogicalPlan = {
+ logDebug(s"Creating new cached data source for $in")
+ val table = client.getTable(in.database, in.name)
+
+ // It does not appear that the ql client for the metastore has a way to enumerate all the
+ // SerDe properties directly...
+ val method = classOf[Table].getDeclaredMethod("getSerdeInfo")
+ method.setAccessible(true)
+ val serdeInfo = method.invoke(table).asInstanceOf[SerDeInfo]
+
+ val resolvedRelation =
+ ResolvedDataSource(
+ hive,
+ table.getProperty("spark.sql.sources.provider"),
+ serdeInfo.getParameters.toMap)
+
+ LogicalRelation(resolvedRelation.relation)
+ }
+ })
+
+ def refreshTable(databaseName: String, tableName: String): Unit = {
+ cachedDataSourceTables.refresh(TableIdent(databaseName, tableName).toLowerCase)
+ }
+
+ def invalidateTable(databaseName: String, tableName: String): Unit = {
+ cachedDataSourceTables.invalidate(TableIdent(databaseName, tableName).toLowerCase)
+ }
+
val caseSensitive: Boolean = false
+ def createDataSourceTable(tableName: String, provider: String, options: Map[String, String]) = {
+ val (dbName, tblName) = processDatabaseAndTableName("default", tableName)
+ val tbl = new Table(dbName, tblName)
+
+ tbl.setProperty("spark.sql.sources.provider", provider)
+ options.foreach { case (key, value) => tbl.setSerdeParam(key, value) }
+
+ // create the table
+ synchronized {
+ client.createTable(tbl, false)
--- End diff --
Yeah, thats a good idea at least for the types of tables you can create now. We should think about options here and if we want to support non-external tables.
Also I assume you meant something like this?
```scala
tbl.setProperty("EXTERNAL", "TRUE")
tbl.setTableType(TableType.EXTERNAL_TABLE)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3752#issuecomment-67884755
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24708/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3752#issuecomment-67876806
[Test build #24708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24708/consoleFull) for PR 3752 at commit [`563ff40`](https://github.com/apache/spark/commit/563ff401c63c7345baf141dd19851d3dd7c9af08).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3752#issuecomment-67884470
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24707/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3752#discussion_r22183385
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -55,8 +56,60 @@ private[hive] class HiveMetastoreCatalog(hive: HiveContext) extends Catalog with
/** Connection to hive metastore. Usages should lock on `this`. */
protected[hive] val client = Hive.get(hive.hiveconf)
+ // TODO: Use this everywhere instead of tuples or databaseName, tableName,.
+ /** A fully qualified identifier for a table (i.e., database.tableName) */
+ case class TableIdent(database: String, name: String) {
+ def toLowerCase = TableIdent(database.toLowerCase, name.toLowerCase)
+ }
+
+ /** A cache of Spark SQL data source tables that have been accessed. */
+ protected[hive] val cachedDataSourceTables = CacheBuilder.newBuilder()
+ .maximumSize(1000)
+ .build(
+ new CacheLoader[TableIdent, LogicalPlan]() {
+ override def load(in: TableIdent): LogicalPlan = {
+ logDebug(s"Creating new cached data source for $in")
+ val table = client.getTable(in.database, in.name)
+
+ // It does not appear that the ql client for the metastore has a way to enumerate all the
+ // SerDe properties directly...
+ val method = classOf[Table].getDeclaredMethod("getSerdeInfo")
--- End diff --
My reasoning here was that table properties might have other things that could conflict with the options that a data source requires. It seems like SerDe properties are scoped to hold just the options that describe how the serialization library should read the data (which seems analogous to our data sources).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/3752#discussion_r22507568
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -55,8 +56,61 @@ private[hive] class HiveMetastoreCatalog(hive: HiveContext) extends Catalog with
/** Connection to hive metastore. Usages should lock on `this`. */
protected[hive] val client = Hive.get(hive.hiveconf)
+ // TODO: Use this everywhere instead of tuples or databaseName, tableName,.
+ /** A fully qualified identifier for a table (i.e., database.tableName) */
+ case class TableIdent(database: String, name: String) {
+ def toLowerCase = TableIdent(database.toLowerCase, name.toLowerCase)
+ }
+
+ /** A cache of Spark SQL data source tables that have been accessed. */
+ protected[hive] val cachedDataSourceTables = CacheBuilder.newBuilder()
+ .maximumSize(1000)
+ .build(
+ new CacheLoader[TableIdent, LogicalPlan]() {
+ override def load(in: TableIdent): LogicalPlan = {
+ logDebug(s"Creating new cached data source for $in")
+ val table = client.getTable(in.database, in.name)
+
+ // It does not appear that the ql client for the metastore has a way to enumerate all the
+ // SerDe properties directly...
+ val options = table.getTTable.getSd.getSerdeInfo.getParameters.toMap
+
+ val resolvedRelation =
+ ResolvedDataSource(
+ hive,
+ table.getProperty("spark.sql.sources.provider"),
+ options)
+
+ LogicalRelation(resolvedRelation.relation)
+ }
+ })
+
+ def refreshTable(databaseName: String, tableName: String): Unit = {
+ cachedDataSourceTables.refresh(TableIdent(databaseName, tableName).toLowerCase)
+ }
+
+ def invalidateTable(databaseName: String, tableName: String): Unit = {
+ cachedDataSourceTables.invalidate(TableIdent(databaseName, tableName).toLowerCase)
+ }
+
val caseSensitive: Boolean = false
+ def createDataSourceTable(tableName: String, provider: String, options: Map[String, String]) = {
+ val (dbName, tblName) = processDatabaseAndTableName("default", tableName)
+ val tbl = new Table(dbName, tblName)
+
+ tbl.setProperty("spark.sql.sources.provider", provider)
+ options.foreach { case (key, value) => tbl.setSerdeParam(key, value) }
--- End diff --
We are using serde properties to store all parameters that will be passed to a relation provider (for creating a relation), right? Probably we can add a comment at here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3752#discussion_r22204038
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -55,8 +56,60 @@ private[hive] class HiveMetastoreCatalog(hive: HiveContext) extends Catalog with
/** Connection to hive metastore. Usages should lock on `this`. */
protected[hive] val client = Hive.get(hive.hiveconf)
+ // TODO: Use this everywhere instead of tuples or databaseName, tableName,.
+ /** A fully qualified identifier for a table (i.e., database.tableName) */
+ case class TableIdent(database: String, name: String) {
+ def toLowerCase = TableIdent(database.toLowerCase, name.toLowerCase)
+ }
+
+ /** A cache of Spark SQL data source tables that have been accessed. */
+ protected[hive] val cachedDataSourceTables = CacheBuilder.newBuilder()
+ .maximumSize(1000)
+ .build(
+ new CacheLoader[TableIdent, LogicalPlan]() {
+ override def load(in: TableIdent): LogicalPlan = {
+ logDebug(s"Creating new cached data source for $in")
+ val table = client.getTable(in.database, in.name)
+
+ // It does not appear that the ql client for the metastore has a way to enumerate all the
+ // SerDe properties directly...
+ val method = classOf[Table].getDeclaredMethod("getSerdeInfo")
+ method.setAccessible(true)
+ val serdeInfo = method.invoke(table).asInstanceOf[SerDeInfo]
+
+ val resolvedRelation =
+ ResolvedDataSource(
+ hive,
+ table.getProperty("spark.sql.sources.provider"),
+ serdeInfo.getParameters.toMap)
+
+ LogicalRelation(resolvedRelation.relation)
+ }
+ })
+
+ def refreshTable(databaseName: String, tableName: String): Unit = {
+ cachedDataSourceTables.refresh(TableIdent(databaseName, tableName).toLowerCase)
+ }
+
+ def invalidateTable(databaseName: String, tableName: String): Unit = {
+ cachedDataSourceTables.invalidate(TableIdent(databaseName, tableName).toLowerCase)
+ }
+
val caseSensitive: Boolean = false
+ def createDataSourceTable(tableName: String, provider: String, options: Map[String, String]) = {
+ val (dbName, tblName) = processDatabaseAndTableName("default", tableName)
+ val tbl = new Table(dbName, tblName)
+
+ tbl.setProperty("spark.sql.sources.provider", provider)
+ options.foreach { case (key, value) => tbl.setSerdeParam(key, value) }
+
+ // create the table
+ synchronized {
+ client.createTable(tbl, false)
--- End diff --
Oh yes. I was reading part of HCatalog code, which is actually manipulating `metastore.api.Table`, whose `setTableType` method accepts a string.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3752#issuecomment-67790092
[Test build #24689 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24689/consoleFull) for PR 3752 at commit [`1002d20`](https://github.com/apache/spark/commit/1002d20e1648e84a0a5e812d87190ac1439cb702).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class TableIdent(database: String, name: String) `
* `case class CreateMetastoreDataSource(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/3752#discussion_r22161200
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -55,8 +56,60 @@ private[hive] class HiveMetastoreCatalog(hive: HiveContext) extends Catalog with
/** Connection to hive metastore. Usages should lock on `this`. */
protected[hive] val client = Hive.get(hive.hiveconf)
+ // TODO: Use this everywhere instead of tuples or databaseName, tableName,.
+ /** A fully qualified identifier for a table (i.e., database.tableName) */
+ case class TableIdent(database: String, name: String) {
+ def toLowerCase = TableIdent(database.toLowerCase, name.toLowerCase)
+ }
+
+ /** A cache of Spark SQL data source tables that have been accessed. */
+ protected[hive] val cachedDataSourceTables = CacheBuilder.newBuilder()
+ .maximumSize(1000)
+ .build(
+ new CacheLoader[TableIdent, LogicalPlan]() {
+ override def load(in: TableIdent): LogicalPlan = {
+ logDebug(s"Creating new cached data source for $in")
+ val table = client.getTable(in.database, in.name)
+
+ // It does not appear that the ql client for the metastore has a way to enumerate all the
+ // SerDe properties directly...
+ val method = classOf[Table].getDeclaredMethod("getSerdeInfo")
--- End diff --
Also I think `SerDeInfo` might not be a proper place to put external data source table options. Semantically, these options are more like general table properties, thus using `table.putProperty` might be better.
@yhuai Are there notable differences between SerDe properties and general table properties in Hive? Or to be more specifically, differences between properties saved in `metastore.Table.getTTable.getParameters` and those in `metastore.Table.getTTable.getSd.getSerdeInfo.getParameters.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3752#issuecomment-67884465
[Test build #24707 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24707/consoleFull) for PR 3752 at commit [`e23f8fb`](https://github.com/apache/spark/commit/e23f8fbd039348e4d2f3c60335a3d60f9c61e19d).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class TableIdent(database: String, name: String) `
* `case class CreateMetastoreDataSource(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3752#issuecomment-67790096
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24689/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/3752#issuecomment-70783358
Let's close it since #3960 has been merged.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4912][SQL] Persistent tables for the Sp...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3752#issuecomment-67750987
[Test build #24677 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24677/consoleFull) for PR 3752 at commit [`055869e`](https://github.com/apache/spark/commit/055869e02d77c1d0f18bc9671e02234e38b26cc2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class TableIdent(database: String, name: String) `
* `case class CreateMetastoreDataSource(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org