You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by bomeng <gi...@git.apache.org> on 2016/03/27 17:49:21 UTC

[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...

GitHub user bomeng opened a pull request:

    https://github.com/apache/spark/pull/11988

    [SPARK-14123] [SQL] DDL: Create / Drop Function

    ## What changes were proposed in this pull request?
    
    This is the first step to support create / drop functions by using catalog based on @yhuai document. Instead of using NativeDDLCommand, we will call the functions in the catalog to do the jobs.
    
    ## How was this patch tested?
    
    I've create a new test suite called DDLSuite that will test the DDL functions, eventually it will contains more DDL test cases, such as Create DB / Table, etc.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/bomeng/spark SPARK-14123

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/11988.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #11988
    
----
commit 033b1acf973b12e764f45118a05cd7bd177aab79
Author: bomeng <bm...@us.ibm.com>
Date:   2016-03-26T22:33:30Z

    ddl: Function related commands

commit bdf44b4e71f180ebf970ef40d5850d558d23a073
Author: bomeng <bm...@us.ibm.com>
Date:   2016-03-27T06:09:43Z

    add test cases and update code style

commit 67425295b389678ecc46b803f8085a71d37c3271
Author: bomeng <bm...@us.ibm.com>
Date:   2016-03-27T15:03:06Z

    merge upstream

commit 3d4761a81c0209d21e1e77656cdb7d6aa5dcda75
Author: bomeng <bm...@us.ibm.com>
Date:   2016-03-27T15:37:47Z

    add drop function support and create a new test case file

commit 7f5325c0a6924aea1f0c75b01e73cc4febe1eb9f
Author: bomeng <bm...@us.ibm.com>
Date:   2016-03-27T15:43:24Z

    update test cases

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the pull request:

    https://github.com/apache/spark/pull/11988#issuecomment-202166458
  
    @bomeng Overall, looks good to me. Just a few minor comments.
    
    Can you also emphasize the following point in the `How was this patch tested?`:
    
    - In this PR, since both `HiveContext` and `SQLContext` share the same implementation of `Create Function` and `Drop Function`, the existing test cases in `HiveContext` also verify the correctness of this PR.
    
    Thanks for your fast work!
    
    cc @yhuai 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11988#discussion_r57536136
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala ---
    @@ -21,15 +21,21 @@ import java.util.NoSuchElementException
     
     import org.apache.spark.internal.Logging
     import org.apache.spark.rdd.RDD
    -import org.apache.spark.sql.{Dataset, Row, SQLContext}
    -import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow, TableIdentifier}
    +import org.apache.spark.sql.catalyst.catalog.CatalogFunction
     import org.apache.spark.sql.catalyst.errors.TreeNodeException
     import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference}
     import org.apache.spark.sql.catalyst.plans.logical
     import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.catalyst.CatalystTypeConverters
    +import org.apache.spark.sql.catalyst.FunctionIdentifier
    +import org.apache.spark.sql.catalyst.InternalRow
    +import org.apache.spark.sql.catalyst.TableIdentifier
     import org.apache.spark.sql.execution.SparkPlan
     import org.apache.spark.sql.internal.SQLConf
     import org.apache.spark.sql.types._
    +import org.apache.spark.sql.Dataset
    +import org.apache.spark.sql.Row
    +import org.apache.spark.sql.SQLContext
    --- End diff --
    
    ```scala
    import org.apache.spark.sql.{Dataset, Row, SQLContext}
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11988#discussion_r57536070
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala ---
    @@ -434,3 +440,57 @@ case class SetDatabaseCommand(databaseName: String) extends RunnableCommand {
     
       override val output: Seq[Attribute] = Seq.empty
     }
    +
    +/**
    +  * A command for users to create a function.
    +  * The syntax of using this command in SQL is
    +  * {{{
    +  *   CREATE TEMPORARY FUNCTION function_name AS class_name;
    +  *   CREATE FUNCTION [db_name.]function_name AS class_name
    +  *     [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
    +  * }}}
    +  */
    +case class CreateFunction(
    +    databaseName: Option[String],
    +    functionName: String, alias: String,
    +    resources: Seq[(String, String)],
    +    isTemp: Boolean)(sql: String) extends RunnableCommand {
    +  override def run(sqlContext: SQLContext): Seq[Row] = {
    +    val catalog = sqlContext.sessionState.catalog
    +    val db = if (databaseName.isDefined) {
    +      databaseName
    +    } else {
    +      Some(catalog.getCurrentDatabase)
    +    }
    --- End diff --
    
    Same rewrite here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11988#discussion_r57536146
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala ---
    @@ -21,15 +21,21 @@ import java.util.NoSuchElementException
     
     import org.apache.spark.internal.Logging
     import org.apache.spark.rdd.RDD
    -import org.apache.spark.sql.{Dataset, Row, SQLContext}
    -import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow, TableIdentifier}
    +import org.apache.spark.sql.catalyst.catalog.CatalogFunction
     import org.apache.spark.sql.catalyst.errors.TreeNodeException
     import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference}
     import org.apache.spark.sql.catalyst.plans.logical
     import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.catalyst.CatalystTypeConverters
    +import org.apache.spark.sql.catalyst.FunctionIdentifier
    +import org.apache.spark.sql.catalyst.InternalRow
    +import org.apache.spark.sql.catalyst.TableIdentifier
    --- End diff --
    
    ```scala
    import org.apache.spark.sql.catalyst.{CatalystTypeConverters, FunctionIdentifier, InternalRow, TableIdentifier}
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11988#discussion_r57536068
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala ---
    @@ -434,3 +440,57 @@ case class SetDatabaseCommand(databaseName: String) extends RunnableCommand {
     
       override val output: Seq[Attribute] = Seq.empty
     }
    +
    +/**
    +  * A command for users to create a function.
    +  * The syntax of using this command in SQL is
    +  * {{{
    +  *   CREATE TEMPORARY FUNCTION function_name AS class_name;
    +  *   CREATE FUNCTION [db_name.]function_name AS class_name
    +  *     [USING JAR|FILE|ARCHIVE 'file_uri' [, JAR|FILE|ARCHIVE 'file_uri'] ];
    +  * }}}
    +  */
    +case class CreateFunction(
    +    databaseName: Option[String],
    +    functionName: String, alias: String,
    +    resources: Seq[(String, String)],
    +    isTemp: Boolean)(sql: String) extends RunnableCommand {
    +  override def run(sqlContext: SQLContext): Seq[Row] = {
    +    val catalog = sqlContext.sessionState.catalog
    +    val db = if (databaseName.isDefined) {
    +      databaseName
    +    } else {
    +      Some(catalog.getCurrentDatabase)
    +    }
    +    val functionIdentifier = FunctionIdentifier(functionName, db)
    +    catalog.createFunction(CatalogFunction(functionIdentifier, alias))
    +    Seq.empty[Row]
    +  }
    +
    +  override val output: Seq[Attribute] = Seq.empty
    +}
    +
    +/**
    +  * The DDL command that drops a function.
    +  * ifExists: returns an error if the function doesn't exist, unless this is true.
    +  * isTemp: indicates if it is a temporary function.
    +  */
    +case class DropFunction(
    +    databaseName: Option[String],
    +    functionName: String,
    +    ifExists: Boolean,
    +    isTemp: Boolean)(sql: String) extends RunnableCommand {
    +  override def run(sqlContext: SQLContext): Seq[Row] = {
    +    val catalog = sqlContext.sessionState.catalog
    +    val db = if (databaseName.isDefined) {
    +      databaseName
    +    } else {
    +      Some(catalog.getCurrentDatabase)
    +    }
    --- End diff --
    
    ```
        val db = databaseName.getOrElse(catalog.getCurrentDatabase)
    ```
    
    Then, use `Some` in the next statement.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11988#issuecomment-202094493
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...

Posted by bomeng <gi...@git.apache.org>.
Github user bomeng commented on the pull request:

    https://github.com/apache/spark/pull/11988#issuecomment-202218950
  
    Hive tests failed. Closed for now for more investigation.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14123] [SQL] DDL: Create / Drop Functio...

Posted by bomeng <gi...@git.apache.org>.
Github user bomeng closed the pull request at:

    https://github.com/apache/spark/pull/11988


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org