You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by yhuai <gi...@git.apache.org> on 2015/02/16 01:41:55 UTC

[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...

GitHub user yhuai opened a pull request:

    https://github.com/apache/spark/pull/4617

    [SPARK-5746][SQL] Check invalid cases for the write path of data source API

    JIRA: https://issues.apache.org/jira/browse/SPARK-5746
    
    @liancheng @marmbrus

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/yhuai/spark insertOverwrite

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/4617.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4617
    
----
commit a88c5168901dc914ee9dee79bce5261b49a7e35e
Author: Yin Huai <yh...@databricks.com>
Date:   2015-02-13T22:57:23Z

    DDLParser will take a parsering function to take care CTAS statements.

commit 6b7545c4abc4455df07b4508cb192718866c8e21
Author: Yin Huai <yh...@databricks.com>
Date:   2015-02-16T00:21:23Z

    Add a pre write check to the data source API.

commit 99da57ea60b5a67978423631e81bba38373a6e06
Author: Yin Huai <yh...@databricks.com>
Date:   2015-02-16T00:37:12Z

    Merge remote-tracking branch 'upstream/master' into insertOverwrite

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...

Posted by marmbrus <gi...@git.apache.org>.

Github user marmbrus commented on a diff in the pull request:

    https://github.com/apache/spark/pull/4617#discussion_r24776541
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/rules.scala ---
    @@ -74,3 +74,68 @@ private[sql] object PreInsertCastAndRename extends Rule[LogicalPlan] {
         }
       }
     }
    +
    +/**
    + * A rule to do various checks before inserting into or writing to a data source table.
    + */
    +private[sql] case class PreWriteCheck(catalog: Catalog) extends Rule[LogicalPlan] {
    +  def failAnalysis(msg: String) = { throw new AnalysisException(msg) }
    +
    +  def apply(plan: LogicalPlan): LogicalPlan = {
    +    plan.foreach {
    +      case i @ logical.InsertIntoTable(
    +        l @ LogicalRelation(t: InsertableRelation), partition, query, overwrite) =>
    +        // Right now, we do not support insert into a data source table with partition specs.
    +        if (partition.nonEmpty) {
    +          failAnalysis(s"Insert into a partition is not allowed because $l is not partitioned.")
    +        } else {
    +          // Get all input data source relations of the query.
    +          val srcRelations = query.collect {
    +            case LogicalRelation(src: BaseRelation) => src
    +          }
    +          if (srcRelations.exists(src => src == t)) {
    +            failAnalysis(
    +              s"It is not allowed to insert overwrite the table because it is used as " +
    +              s"an input table.")
    +          } else {
    +            // OK
    +          }
    +        }
    +
    +      case i @ logical.InsertIntoTable(l: LogicalRelation, partition, query, overwrite) =>
    --- End diff --
    
    Make the ! instanceOf here explicit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/4617#issuecomment-74452655
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27535/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-5746][SQL] Check invalid cases for the ...

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/4617


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org