You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by bomeng <gi...@git.apache.org> on 2016/03/27 17:49:21 UTC
[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...
GitHub user bomeng opened a pull request:
https://github.com/apache/spark/pull/11988
[SPARK-14123] [SQL] DDL: Create / Drop Function
## What changes were proposed in this pull request?
This is the first step to support create / drop functions by using catalog based on @yhuai document. Instead of using NativeDDLCommand, we will call the functions in the catalog to do the jobs.
## How was this patch tested?
I've create a new test suite called DDLSuite that will test the DDL functions, eventually it will contains more DDL test cases, such as Create DB / Table, etc.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bomeng/spark SPARK-14123
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11988.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11988
----
commit 033b1acf973b12e764f45118a05cd7bd177aab79
Author: bomeng <bm...@us.ibm.com>
Date: 2016-03-26T22:33:30Z
ddl: Function related commands
commit bdf44b4e71f180ebf970ef40d5850d558d23a073
Author: bomeng <bm...@us.ibm.com>
Date: 2016-03-27T06:09:43Z
add test cases and update code style
commit 67425295b389678ecc46b803f8085a71d37c3271
Author: bomeng <bm...@us.ibm.com>
Date: 2016-03-27T15:03:06Z
merge upstream
commit 3d4761a81c0209d21e1e77656cdb7d6aa5dcda75
Author: bomeng <bm...@us.ibm.com>
Date: 2016-03-27T15:37:47Z
add drop function support and create a new test case file
commit 7f5325c0a6924aea1f0c75b01e73cc4febe1eb9f
Author: bomeng <bm...@us.ibm.com>
Date: 2016-03-27T15:43:24Z
update test cases
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/11988#issuecomment-202166458
@bomeng Overall, looks good to me. Just a few minor comments.
Can you also emphasize the following point in the `How was this patch tested?`:
- In this PR, since both `HiveContext` and `SQLContext` share the same implementation of `Create Function` and `Drop Function`, the existing test cases in `HiveContext` also verify the correctness of this PR.
Thanks for your fast work!
cc @yhuai
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11988#discussion_r57536136
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala ---
@@ -21,15 +21,21 @@ import java.util.NoSuchElementException
import org.apache.spark.internal.Logging
import org.apache.spark.rdd.RDD
-import org.apache.spark.sql.{Dataset, Row, SQLContext}
-import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow, TableIdentifier}
+import org.apache.spark.sql.catalyst.catalog.CatalogFunction
import org.apache.spark.sql.catalyst.errors.TreeNodeException
import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference}
import org.apache.spark.sql.catalyst.plans.logical
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.CatalystTypeConverters
+import org.apache.spark.sql.catalyst.FunctionIdentifier
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.execution.SparkPlan
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.SQLContext
--- End diff --
```scala
import org.apache.spark.sql.{Dataset, Row, SQLContext}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11988#discussion_r57536070
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala ---
@@ -434,3 +440,57 @@ case class SetDatabaseCommand(databaseName: String) extends RunnableCommand {
override val output: Seq[Attribute] = Seq.empty
}
+
+/**
+ * A command for users to create a function.
+ * The syntax of using this command in SQL is
+ * {{{
+ * CREATE TEMPORARY FUNCTION function_name AS class_name;
+ * CREATE FUNCTION [db_name.]function_name AS class_name
+ * [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
+ * }}}
+ */
+case class CreateFunction(
+ databaseName: Option[String],
+ functionName: String, alias: String,
+ resources: Seq[(String, String)],
+ isTemp: Boolean)(sql: String) extends RunnableCommand {
+ override def run(sqlContext: SQLContext): Seq[Row] = {
+ val catalog = sqlContext.sessionState.catalog
+ val db = if (databaseName.isDefined) {
+ databaseName
+ } else {
+ Some(catalog.getCurrentDatabase)
+ }
--- End diff --
Same rewrite here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11988#discussion_r57536146
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala ---
@@ -21,15 +21,21 @@ import java.util.NoSuchElementException
import org.apache.spark.internal.Logging
import org.apache.spark.rdd.RDD
-import org.apache.spark.sql.{Dataset, Row, SQLContext}
-import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow, TableIdentifier}
+import org.apache.spark.sql.catalyst.catalog.CatalogFunction
import org.apache.spark.sql.catalyst.errors.TreeNodeException
import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference}
import org.apache.spark.sql.catalyst.plans.logical
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.CatalystTypeConverters
+import org.apache.spark.sql.catalyst.FunctionIdentifier
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.TableIdentifier
--- End diff --
```scala
import org.apache.spark.sql.catalyst.{CatalystTypeConverters, FunctionIdentifier, InternalRow, TableIdentifier}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11988#discussion_r57536068
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala ---
@@ -434,3 +440,57 @@ case class SetDatabaseCommand(databaseName: String) extends RunnableCommand {
override val output: Seq[Attribute] = Seq.empty
}
+
+/**
+ * A command for users to create a function.
+ * The syntax of using this command in SQL is
+ * {{{
+ * CREATE TEMPORARY FUNCTION function_name AS class_name;
+ * CREATE FUNCTION [db_name.]function_name AS class_name
+ * [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
+ * }}}
+ */
+case class CreateFunction(
+ databaseName: Option[String],
+ functionName: String, alias: String,
+ resources: Seq[(String, String)],
+ isTemp: Boolean)(sql: String) extends RunnableCommand {
+ override def run(sqlContext: SQLContext): Seq[Row] = {
+ val catalog = sqlContext.sessionState.catalog
+ val db = if (databaseName.isDefined) {
+ databaseName
+ } else {
+ Some(catalog.getCurrentDatabase)
+ }
+ val functionIdentifier = FunctionIdentifier(functionName, db)
+ catalog.createFunction(CatalogFunction(functionIdentifier, alias))
+ Seq.empty[Row]
+ }
+
+ override val output: Seq[Attribute] = Seq.empty
+}
+
+/**
+ * The DDL command that drops a function.
+ * ifExists: returns an error if the function doesn't exist, unless this is true.
+ * isTemp: indicates if it is a temporary function.
+ */
+case class DropFunction(
+ databaseName: Option[String],
+ functionName: String,
+ ifExists: Boolean,
+ isTemp: Boolean)(sql: String) extends RunnableCommand {
+ override def run(sqlContext: SQLContext): Seq[Row] = {
+ val catalog = sqlContext.sessionState.catalog
+ val db = if (databaseName.isDefined) {
+ databaseName
+ } else {
+ Some(catalog.getCurrentDatabase)
+ }
--- End diff --
```
val db = databaseName.getOrElse(catalog.getCurrentDatabase)
```
Then, use `Some` in the next statement.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11988#issuecomment-202094493
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...
Posted by bomeng <gi...@git.apache.org>.
Github user bomeng commented on the pull request:
https://github.com/apache/spark/pull/11988#issuecomment-202218950
Hive tests failed. Closed for now for more investigation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...
Posted by bomeng <gi...@git.apache.org>.
Github user bomeng closed the pull request at:
https://github.com/apache/spark/pull/11988
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org