You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by rdblue <gi...@git.apache.org> on 2018/08/02 21:38:17 UTC
[GitHub] spark pull request #21978: SPARK-25006: Add CatalogTableIdentifier.
GitHub user rdblue opened a pull request:
https://github.com/apache/spark/pull/21978
SPARK-25006: Add CatalogTableIdentifier.
## What changes were proposed in this pull request?
This adds `CatalogTableIdentifier`, which is an identifier that consists of a triple: catalog, database, and table. Catalog and database are optional.
The existing `TableIdentifier` class extends `CatalogTableIdentifier` and is guarateed to have no defined catalog component. Classes that expect a `TableIdentifier` will continue to use `TableIdentifier` to ensure that catalogs are not leaked into code paths that do not support them.
This adds a parser rule, `catalogTableIdentifier`, that can parse identifiers with a catalog. An identifier with only two components will match database and table, leaving the catalog undefined. Only identifiers with three components will have a defined catalog. In addition, rules must be re-written to support `catalogTableIdentifier`. Existing rules will continue to use `tableIdentifier` with no catalog.
## How was this patch tested?
Existing tests. This should not change any behavior.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rdblue/spark SPARK-25006-add-catalog-to-table-identifier
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21978.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21978
----
commit d61e0f1ccdce630d65c81ee27a233a14759415ea
Author: Ryan Blue <bl...@...>
Date: 2018-08-02T21:03:21Z
SPARK-25006: Add CatalogTableIdentifier.
This adds CatalogTableIdentifier, which is an identifier that consists
of a triple: catalog, database, and table. Catalog and database are
optional.
The existing TableIdentifier class extends CatalogTableIdentifier and is
guarateed to have no defined catalog component. Classes that expect a
TableIdentifier should continue to use TableIdentifier to ensure that
catalogs are not leaked into code paths that do not support them.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94075 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94075/testReport)** for PR 21978 at commit [`6fe2d07`](https://github.com/apache/spark/commit/6fe2d07de57feff8903f9c91eb0841eaa4646505).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94163 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94163/testReport)** for PR 21978 at commit [`6fe2d07`](https://github.com/apache/spark/commit/6fe2d07de57feff8903f9c91eb0841eaa4646505).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class UnresolvedRelation(table: CatalogTableIdentifier) extends LeafNode `
* `sealed trait IdentifierWithOptionalDatabaseAndCatalog `
* `case class CatalogTableIdentifier(table: String, database: Option[String], catalog: Option[String])`
* `class TableIdentifier(name: String, db: Option[String])`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94188 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94188/testReport)** for PR 21978 at commit [`00295ee`](https://github.com/apache/spark/commit/00295ee6b3713995641c90a9b3b7cd4a6b79ded6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1783/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94070 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94070/testReport)** for PR 21978 at commit [`f34e23b`](https://github.com/apache/spark/commit/f34e23b4c427346344805a2076d960b5f74d2f31).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94068 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94068/testReport)** for PR 21978 at commit [`7abc8b8`](https://github.com/apache/spark/commit/7abc8b8df1cae4fe91601cca3b9565e952511318).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94068/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1768/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94068 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94068/testReport)** for PR 21978 at commit [`7abc8b8`](https://github.com/apache/spark/commit/7abc8b8df1cae4fe91601cca3b9565e952511318).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class UnresolvedRelation(table: CatalogTableIdentifier) extends LeafNode `
* `sealed trait IdentifierWithOptionalDatabaseAndCatalog `
* `case class CatalogTableIdentifier(table: String, database: Option[String], catalog: Option[String])`
* `class TableIdentifier(name: String, db: Option[String])`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: [SPARK-25006][SQL] Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: [SPARK-25006][SQL] Add CatalogTableIdentifier.
Posted by mccheah <gi...@git.apache.org>.
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21978
Wanted to follow up here - are we planning on merging this or are there more things we need to discuss?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1704/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21978
@cloud-fan, that's fine with me since #17185 is already merged. Would this conflict with #17185? We can just add a case that detects whether the first identifier in the seq is a catalog when updating expressions.
This PR is just the start for adding catalog to table identifiers. None of the SQL statements are modified in this PR on purpose: code paths will need to be updated to support `CatalogTableIdentifier`. This introduces the class so we can use type safety to ensure that `CatalogTableIdentifier` doens't leak into the code paths that don't support it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1713/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94163/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21978: [SPARK-25006][SQL] Add CatalogTableIdentifier.
Posted by mccheah <gi...@git.apache.org>.
Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/21978#discussion_r237578805
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/identifiers.scala ---
@@ -18,48 +18,106 @@
package org.apache.spark.sql.catalyst
/**
- * An identifier that optionally specifies a database.
+ * An identifier that optionally specifies a database and catalog.
*
* Format (unquoted): "name" or "db.name"
--- End diff --
Update formats in these scaladocs.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21978: [SPARK-25006][SQL] Add CatalogTableIdentifier.
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/21978#discussion_r237660050
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/identifiers.scala ---
@@ -18,48 +18,106 @@
package org.apache.spark.sql.catalyst
/**
- * An identifier that optionally specifies a database.
+ * An identifier that optionally specifies a database and catalog.
*
* Format (unquoted): "name" or "db.name"
* Format (quoted): "`name`" or "`db`.`name`"
*/
-sealed trait IdentifierWithDatabase {
+sealed trait IdentifierWithOptionalDatabaseAndCatalog {
val identifier: String
def database: Option[String]
+ def catalog: Option[String]
+
/*
* Escapes back-ticks within the identifier name with double-back-ticks.
*/
private def quoteIdentifier(name: String): String = name.replace("`", "``")
def quotedString: String = {
- val replacedId = quoteIdentifier(identifier)
- val replacedDb = database.map(quoteIdentifier(_))
-
- if (replacedDb.isDefined) s"`${replacedDb.get}`.`$replacedId`" else s"`$replacedId`"
+ // database is required if catalog is present
+ assert(database.isDefined || catalog.isEmpty)
+ def q(s: String): String = s"`${quoteIdentifier(s)}`"
+ Seq(catalog.map(q), database.map(q), Some(q(identifier))).flatten.mkString(".")
}
def unquotedString: String = {
- if (database.isDefined) s"${database.get}.$identifier" else identifier
+ Seq(catalog, database, Some(identifier)).flatten.mkString(".")
}
override def toString: String = quotedString
}
+object CatalogTableIdentifier {
+ def apply(table: String): CatalogTableIdentifier =
+ new CatalogTableIdentifier(table, None, None)
+
+ def apply(table: String, database: String): CatalogTableIdentifier =
+ new CatalogTableIdentifier(table, Some(database), None)
+
+ def apply(table: String, database: String, catalog: String): CatalogTableIdentifier =
+ new CatalogTableIdentifier(table, Some(database), Some(catalog))
+}
+
/**
- * Identifies a table in a database.
- * If `database` is not defined, the current database is used.
- * When we register a permanent function in the FunctionRegistry, we use
- * unquotedString as the function name.
+ * Identifies a table in a database and catalog.
+ * If `database` is not defined, the current catalog's default database is used.
+ * If `catalog` is not defined, the current catalog is used.
--- End diff --
Agreed. This introduces the ability to expose a catalog to Spark. It doesn't actually add any user-facing operations.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21978
Retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21978
FYI @jzhuge
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1709/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21978
@gatorsmile and @cloud-fan, this adds catalog to `TableIdentifier` in preparation for multi-catalog support. `TableIdentifier` continues to work as-is to ensure that there are no behavior changes in code paths that do not have catalog support. I've updated `UnresolvedRelation` to demonstrate how migration to `CatalogTableIdentifier` will work.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1815/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21978
Retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94188 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94188/testReport)** for PR 21978 at commit [`00295ee`](https://github.com/apache/spark/commit/00295ee6b3713995641c90a9b3b7cd4a6b79ded6).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class UnresolvedRelation(table: CatalogTableIdentifier) extends LeafNode `
* `sealed trait IdentifierWithOptionalDatabaseAndCatalog `
* `case class CatalogTableIdentifier(table: String, database: Option[String], catalog: Option[String])`
* `class TableIdentifier(table: String, db: Option[String])`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94375 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94375/testReport)** for PR 21978 at commit [`00295ee`](https://github.com/apache/spark/commit/00295ee6b3713995641c90a9b3b7cd4a6b79ded6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class UnresolvedRelation(table: CatalogTableIdentifier) extends LeafNode `
* `sealed trait IdentifierWithOptionalDatabaseAndCatalog `
* `case class CatalogTableIdentifier(table: String, database: Option[String], catalog: Option[String])`
* `class TableIdentifier(table: String, db: Option[String])`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21978: [SPARK-25006][SQL] Add CatalogTableIdentifier.
Posted by mccheah <gi...@git.apache.org>.
Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/21978#discussion_r237578911
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/identifiers.scala ---
@@ -18,48 +18,106 @@
package org.apache.spark.sql.catalyst
/**
- * An identifier that optionally specifies a database.
+ * An identifier that optionally specifies a database and catalog.
*
* Format (unquoted): "name" or "db.name"
* Format (quoted): "`name`" or "`db`.`name`"
*/
-sealed trait IdentifierWithDatabase {
+sealed trait IdentifierWithOptionalDatabaseAndCatalog {
val identifier: String
def database: Option[String]
+ def catalog: Option[String]
+
/*
* Escapes back-ticks within the identifier name with double-back-ticks.
*/
private def quoteIdentifier(name: String): String = name.replace("`", "``")
def quotedString: String = {
- val replacedId = quoteIdentifier(identifier)
- val replacedDb = database.map(quoteIdentifier(_))
-
- if (replacedDb.isDefined) s"`${replacedDb.get}`.`$replacedId`" else s"`$replacedId`"
+ // database is required if catalog is present
+ assert(database.isDefined || catalog.isEmpty)
+ def q(s: String): String = s"`${quoteIdentifier(s)}`"
+ Seq(catalog.map(q), database.map(q), Some(q(identifier))).flatten.mkString(".")
}
def unquotedString: String = {
- if (database.isDefined) s"${database.get}.$identifier" else identifier
+ Seq(catalog, database, Some(identifier)).flatten.mkString(".")
}
override def toString: String = quotedString
}
+object CatalogTableIdentifier {
+ def apply(table: String): CatalogTableIdentifier =
+ new CatalogTableIdentifier(table, None, None)
+
+ def apply(table: String, database: String): CatalogTableIdentifier =
+ new CatalogTableIdentifier(table, Some(database), None)
+
+ def apply(table: String, database: String, catalog: String): CatalogTableIdentifier =
+ new CatalogTableIdentifier(table, Some(database), Some(catalog))
+}
+
/**
- * Identifies a table in a database.
- * If `database` is not defined, the current database is used.
- * When we register a permanent function in the FunctionRegistry, we use
- * unquotedString as the function name.
+ * Identifies a table in a database and catalog.
+ * If `database` is not defined, the current catalog's default database is used.
+ * If `catalog` is not defined, the current catalog is used.
--- End diff --
"current" meaning "global"?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: [SPARK-25006][SQL] Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #99480 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99480/testReport)** for PR 21978 at commit [`beebccf`](https://github.com/apache/spark/commit/beebccf8acbaa1a8a14d6a256aa5ef0b7efefcec).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class UnresolvedRelation(table: CatalogTableIdentifier) extends LeafNode `
* `sealed trait IdentifierWithOptionalDatabaseAndCatalog `
* `case class CatalogTableIdentifier(table: String, database: Option[String], catalog: Option[String])`
* `class TableIdentifier(table: String, db: Option[String])`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: [SPARK-25006][SQL] Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #99480 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99480/testReport)** for PR 21978 at commit [`beebccf`](https://github.com/apache/spark/commit/beebccf8acbaa1a8a14d6a256aa5ef0b7efefcec).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1701/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94064 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94064/testReport)** for PR 21978 at commit [`c0683a8`](https://github.com/apache/spark/commit/c0683a8da656cfededf3f85a00e1357c6e83e0f0).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94061 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94061/testReport)** for PR 21978 at commit [`d61e0f1`](https://github.com/apache/spark/commit/d61e0f1ccdce630d65c81ee27a233a14759415ea).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21978
@cloud-fan, on the dev thread about 2.4 you talked about getting this PR in. What do we need to do next?
I can call a vote on the SPIP if you think that's ready. I just bumped the thread. I'm also interested in getting review comments on this or #21978 if you have any.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94163 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94163/testReport)** for PR 21978 at commit [`6fe2d07`](https://github.com/apache/spark/commit/6fe2d07de57feff8903f9c91eb0841eaa4646505).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21978
I'd like to wait for https://github.com/apache/spark/pull/17185
#17185 allows users to do `db1.table1.col1`, and we can later extend it to `catalog1.db1.table1.col1`.
We should also update the column resolution logic to consider catalog name.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: [SPARK-25006][SQL] Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5546/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: [SPARK-25006][SQL] Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99480/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94064 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94064/testReport)** for PR 21978 at commit [`c0683a8`](https://github.com/apache/spark/commit/c0683a8da656cfededf3f85a00e1357c6e83e0f0).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class UnresolvedRelation(table: CatalogTableIdentifier) extends LeafNode `
* `sealed trait IdentifierWithOptionalDatabaseAndCatalog `
* `case class CatalogTableIdentifier(table: String, database: Option[String], catalog: Option[String])`
* `class TableIdentifier(name: String, db: Option[String])`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94075/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: [SPARK-25006][SQL] Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21978
@cloud-fan, when do you think we can get this in? It doesn't need to go in 2.4 because it doesn't change any read or write paths -- nothing uses CatalogTableIdentifier yet -- but it would be great to get it into master so we can start building paths that do support CatalogTableIdentifier.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21978: [SPARK-25006][SQL] Add CatalogTableIdentifier.
Posted by mccheah <gi...@git.apache.org>.
Github user mccheah commented on a diff in the pull request:
https://github.com/apache/spark/pull/21978#discussion_r237656841
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/identifiers.scala ---
@@ -18,48 +18,106 @@
package org.apache.spark.sql.catalyst
/**
- * An identifier that optionally specifies a database.
+ * An identifier that optionally specifies a database and catalog.
*
* Format (unquoted): "name" or "db.name"
* Format (quoted): "`name`" or "`db`.`name`"
*/
-sealed trait IdentifierWithDatabase {
+sealed trait IdentifierWithOptionalDatabaseAndCatalog {
val identifier: String
def database: Option[String]
+ def catalog: Option[String]
+
/*
* Escapes back-ticks within the identifier name with double-back-ticks.
*/
private def quoteIdentifier(name: String): String = name.replace("`", "``")
def quotedString: String = {
- val replacedId = quoteIdentifier(identifier)
- val replacedDb = database.map(quoteIdentifier(_))
-
- if (replacedDb.isDefined) s"`${replacedDb.get}`.`$replacedId`" else s"`$replacedId`"
+ // database is required if catalog is present
+ assert(database.isDefined || catalog.isEmpty)
+ def q(s: String): String = s"`${quoteIdentifier(s)}`"
+ Seq(catalog.map(q), database.map(q), Some(q(identifier))).flatten.mkString(".")
}
def unquotedString: String = {
- if (database.isDefined) s"${database.get}.$identifier" else identifier
+ Seq(catalog, database, Some(identifier)).flatten.mkString(".")
}
override def toString: String = quotedString
}
+object CatalogTableIdentifier {
+ def apply(table: String): CatalogTableIdentifier =
+ new CatalogTableIdentifier(table, None, None)
+
+ def apply(table: String, database: String): CatalogTableIdentifier =
+ new CatalogTableIdentifier(table, Some(database), None)
+
+ def apply(table: String, database: String, catalog: String): CatalogTableIdentifier =
+ new CatalogTableIdentifier(table, Some(database), Some(catalog))
+}
+
/**
- * Identifies a table in a database.
- * If `database` is not defined, the current database is used.
- * When we register a permanent function in the FunctionRegistry, we use
- * unquotedString as the function name.
+ * Identifies a table in a database and catalog.
+ * If `database` is not defined, the current catalog's default database is used.
+ * If `catalog` is not defined, the current catalog is used.
--- End diff --
Sounds good. When we add the logical side of leveraging catalogs we can revisit the API of how to set the current catalog.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94232 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94232/testReport)** for PR 21978 at commit [`00295ee`](https://github.com/apache/spark/commit/00295ee6b3713995641c90a9b3b7cd4a6b79ded6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1707/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94075 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94075/testReport)** for PR 21978 at commit [`6fe2d07`](https://github.com/apache/spark/commit/6fe2d07de57feff8903f9c91eb0841eaa4646505).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class UnresolvedRelation(table: CatalogTableIdentifier) extends LeafNode `
* `sealed trait IdentifierWithOptionalDatabaseAndCatalog `
* `case class CatalogTableIdentifier(table: String, database: Option[String], catalog: Option[String])`
* `class TableIdentifier(name: String, db: Option[String])`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94232/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21978: [SPARK-25006][SQL] Add CatalogTableIdentifier.
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/21978#discussion_r237585203
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/identifiers.scala ---
@@ -18,48 +18,106 @@
package org.apache.spark.sql.catalyst
/**
- * An identifier that optionally specifies a database.
+ * An identifier that optionally specifies a database and catalog.
*
* Format (unquoted): "name" or "db.name"
* Format (quoted): "`name`" or "`db`.`name`"
*/
-sealed trait IdentifierWithDatabase {
+sealed trait IdentifierWithOptionalDatabaseAndCatalog {
val identifier: String
def database: Option[String]
+ def catalog: Option[String]
+
/*
* Escapes back-ticks within the identifier name with double-back-ticks.
*/
private def quoteIdentifier(name: String): String = name.replace("`", "``")
def quotedString: String = {
- val replacedId = quoteIdentifier(identifier)
- val replacedDb = database.map(quoteIdentifier(_))
-
- if (replacedDb.isDefined) s"`${replacedDb.get}`.`$replacedId`" else s"`$replacedId`"
+ // database is required if catalog is present
+ assert(database.isDefined || catalog.isEmpty)
+ def q(s: String): String = s"`${quoteIdentifier(s)}`"
+ Seq(catalog.map(q), database.map(q), Some(q(identifier))).flatten.mkString(".")
}
def unquotedString: String = {
- if (database.isDefined) s"${database.get}.$identifier" else identifier
+ Seq(catalog, database, Some(identifier)).flatten.mkString(".")
}
override def toString: String = quotedString
}
+object CatalogTableIdentifier {
+ def apply(table: String): CatalogTableIdentifier =
+ new CatalogTableIdentifier(table, None, None)
+
+ def apply(table: String, database: String): CatalogTableIdentifier =
+ new CatalogTableIdentifier(table, Some(database), None)
+
+ def apply(table: String, database: String, catalog: String): CatalogTableIdentifier =
+ new CatalogTableIdentifier(table, Some(database), Some(catalog))
+}
+
/**
- * Identifies a table in a database.
- * If `database` is not defined, the current database is used.
- * When we register a permanent function in the FunctionRegistry, we use
- * unquotedString as the function name.
+ * Identifies a table in a database and catalog.
+ * If `database` is not defined, the current catalog's default database is used.
+ * If `catalog` is not defined, the current catalog is used.
--- End diff --
No, we want to move away from a special global catalog. I think that Spark should have a current catalog, like a current database, which is used to resolve references that don't have an explicit catalog. That would have a default, just like the current database has a default.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94375 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94375/testReport)** for PR 21978 at commit [`00295ee`](https://github.com/apache/spark/commit/00295ee6b3713995641c90a9b3b7cd4a6b79ded6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94064/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94070/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94375/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94061 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94061/testReport)** for PR 21978 at commit [`d61e0f1`](https://github.com/apache/spark/commit/d61e0f1ccdce630d65c81ee27a233a14759415ea).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class UnresolvedRelation(table: CatalogTableIdentifier) extends LeafNode `
* `sealed trait IdentifierWithOptionalDatabaseAndCatalog `
* `case class CatalogTableIdentifier(table: String, database: Option[String], catalog: Option[String])`
* `class TableIdentifier(name: String, db: Option[String])`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94061/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: [SPARK-25006][SQL] Add CatalogTableIdentifier.
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21978
Rebased on master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94070 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94070/testReport)** for PR 21978 at commit [`f34e23b`](https://github.com/apache/spark/commit/f34e23b4c427346344805a2076d960b5f74d2f31).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class UnresolvedRelation(table: CatalogTableIdentifier) extends LeafNode `
* `sealed trait IdentifierWithOptionalDatabaseAndCatalog `
* `case class CatalogTableIdentifier(table: String, database: Option[String], catalog: Option[String])`
* `class TableIdentifier(name: String, db: Option[String])`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21978
**[Test build #94232 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94232/testReport)** for PR 21978 at commit [`00295ee`](https://github.com/apache/spark/commit/00295ee6b3713995641c90a9b3b7cd4a6b79ded6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class UnresolvedRelation(table: CatalogTableIdentifier) extends LeafNode `
* `sealed trait IdentifierWithOptionalDatabaseAndCatalog `
* `case class CatalogTableIdentifier(table: String, database: Option[String], catalog: Option[String])`
* `class TableIdentifier(table: String, db: Option[String])`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21978
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94188/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org