You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by rdblue <gi...@git.apache.org> on 2018/02/02 21:36:14 UTC

[GitHub] spark pull request #20488: [SPARK-23321][SQL]: Validate datasource v2 writes

GitHub user rdblue opened a pull request:

    https://github.com/apache/spark/pull/20488

    [SPARK-23321][SQL]: Validate datasource v2 writes

    ## What changes were proposed in this pull request?
    
    DataSourceV2 does not currently apply any validation rules when writing. Other write paths attempt to validate that a data frame can be written to a target table or path and these changes add the same logic to v2.
    
    This updates the logical plan to use InsertIntoTable and applies the insert preprocess rules to writes. It also adds a conversion rule from InserIntoTable to DataSourceV2Write because InsertIntoTable cannot be used in logical plans after analysis.
    
    InsertIntoTable is not necessarily the right logical plan. It assumes that the table exists and can report its schema.
    
    ## How was this patch tested?
    
    Added a test that fails analysis in the preprocess rule.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/rdblue/spark SPARK-23321-validate-datasource-v2-writes

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20488.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20488
    
----
commit 62c569672083c0fa633da1d6edaba40d0bb05819
Author: Ryan Blue <bl...@...>
Date:   2018-01-17T21:58:12Z

    SPARK-22386: DataSourceV2: Use immutable logical plans.

commit f0bd45d3c931941b8092cdac738cb29954e0acdd
Author: Ryan Blue <bl...@...>
Date:   2018-01-24T19:34:42Z

    SPARK-23203: Fix scala style check.

commit 2fdeb4556cd22a092630b341a22a16a59e377183
Author: Ryan Blue <bl...@...>
Date:   2018-01-24T19:54:10Z

    SPARK-23203: Fix Kafka tests, use StreamingDataSourceV2Relation.
    
    This also removes unused imports.

commit ab945a19efe666c41deae9c044002f3455220c1d
Author: Ryan Blue <bl...@...>
Date:   2018-02-02T20:30:33Z

    SPARK-23204: DataFrameReader: Remove v2 table identifier parsing.

commit 3580daf15497a1d49112a0eddd556f74b9b3e280
Author: Ryan Blue <bl...@...>
Date:   2018-02-02T19:04:23Z

    SPARK-23321: Apply preprocess insert rules to DataSourceV2.
    
    This updates the DataSourceV2 write path to use DataSourceV2Relation and
    InsertIntoTable to apply the insert preprocess rules.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #20488: [SPARK-23321][SQL]: Validate datasource v2 writes

Posted by rdblue <gi...@git.apache.org>.

Github user rdblue closed the pull request at:

    https://github.com/apache/spark/pull/20488


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20488: [SPARK-23321][SQL]: Validate datasource v2 writes

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20488
  
    **[Test build #87006 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87006/testReport)** for PR 20488 at commit [`3580daf`](https://github.com/apache/spark/commit/3580daf15497a1d49112a0eddd556f74b9b3e280).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `case class NoopCommand() extends RunnableCommand `


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org