You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gengliangwang <gi...@git.apache.org> on 2017/12/19 14:37:25 UTC

[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

GitHub user gengliangwang opened a pull request:

    https://github.com/apache/spark/pull/20020

    [SPARK-22834][SQL] Make insertion commands have real children to fix UI issues

    ## What changes were proposed in this pull request?
    
    With #19474,  children of insertion commands are missing in UI. 
    To fix it:
    1. Create a new physical plan `DataWritingCommandExec` to exec `DataWritingCommand` with children.  So that the other commands won't be affected.
    2. On creation of `DataWritingCommand`, a new field `allColumns` must be specified, which is the output of analyzed plan. 
    3. In `FileFormatWriter`, the output schema will use `allColumns` instead of the output of optimized plan.
    
    Before code changes:
    ![2017-12-19 10 27 10](https://user-images.githubusercontent.com/1097932/34161850-d2fd0acc-e50c-11e7-898a-177154fe7d8e.png)
    
    After code changes:
    ![2017-12-19 10 27 04](https://user-images.githubusercontent.com/1097932/34161865-de23de26-e50c-11e7-9131-0c32f7b7b749.png)
    
    
    ## How was this patch tested?
    Unit test


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/gengliangwang/spark insert

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20020.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20020
    
----
commit d354895031ba42c28ee0d8665f67ccc6e3766821
Author: Wang Gengliang <lt...@gmail.com>
Date:   2017-12-19T14:16:48Z

    SPARK-22834: Make insert commands have real children to fix UI issues

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85225 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85225/testReport)** for PR 20020 at commit [`7ccfd90`](https://github.com/apache/spark/commit/7ccfd909a71a531be25ab0eccdf7b347d06c2c0a).
     * This patch **fails PySpark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85159 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85159/testReport)** for PR 20020 at commit [`e25a9eb`](https://github.com/apache/spark/commit/e25a9eb285d56a771a56b77534413be59b9f111b).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `trait DataWritingCommand extends Command `
      * `case class DataWritingCommandExec(cmd: DataWritingCommand, children: Seq[SparkPlan])`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85272 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85272/testReport)** for PR 20020 at commit [`f76bef7`](https://github.com/apache/spark/commit/f76bef78658054c160d18611029c6c3f283876af).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85493 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85493/testReport)** for PR 20020 at commit [`18ec016`](https://github.com/apache/spark/commit/18ec01638b9da7f8150e3ea35c2876d6d1f41f3d).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85268/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85396/testReport)** for PR 20020 at commit [`cd2bbf8`](https://github.com/apache/spark/commit/cd2bbf8434a6b142f89f427db8654aeef36cec11).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85266 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85266/testReport)** for PR 20020 at commit [`83e0fba`](https://github.com/apache/spark/commit/83e0fba0574cf41717f6d9557df17587e27b5301).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85396/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158025766
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/DataWritingCommand.scala ---
    @@ -20,30 +20,18 @@ package org.apache.spark.sql.execution.command
     import org.apache.hadoop.conf.Configuration
     
     import org.apache.spark.SparkContext
    -import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.{Row, SparkSession}
    +import org.apache.spark.sql.catalyst.plans.logical.{Command, LogicalPlan}
    +import org.apache.spark.sql.execution.SparkPlan
     import org.apache.spark.sql.execution.datasources.BasicWriteJobStatsTracker
     import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
     import org.apache.spark.util.SerializableConfiguration
     
    -
     /**
      * A special `RunnableCommand` which writes data out and updates metrics.
      */
    -trait DataWritingCommand extends RunnableCommand {
    -
    -  /**
    -   * The input query plan that produces the data to be written.
    -   */
    -  def query: LogicalPlan
    -
    -  // We make the input `query` an inner child instead of a child in order to hide it from the
    -  // optimizer. This is because optimizer may not preserve the output schema names' case, and we
    -  // have to keep the original analyzed plan here so that we can pass the corrected schema to the
    -  // writer. The schema of analyzed plan is what user expects(or specifies), so we should respect
    -  // it when writing.
    -  override protected def innerChildren: Seq[LogicalPlan] = query :: Nil
    --- End diff --
    
    now shall we define query as a child here?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85266 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85266/testReport)** for PR 20020 at commit [`83e0fba`](https://github.com/apache/spark/commit/83e0fba0574cf41717f6d9557df17587e27b5301).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158019546
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala ---
    @@ -87,6 +87,51 @@ case class ExecutedCommandExec(cmd: RunnableCommand) extends LeafExecNode {
       }
     }
     
    +/**
    + * A physical operator that executes the run method of a `RunnableCommand` and
    + * saves the result to prevent multiple executions.
    + *
    + * @param cmd the `RunnableCommand` this operator will run.
    + * @param children the children physical plans ran by the `RunnableCommand`.
    + */
    +case class DataWritingCommandExec(cmd: DataWritingCommand, children: Seq[SparkPlan])
    +  extends SparkPlan {
    +
    +  override lazy val metrics: Map[String, SQLMetric] = cmd.metrics
    +
    +  /**
    +   * A concrete command should override this lazy field to wrap up any side effects caused by the
    +   * command or any other computation that should be evaluated exactly once. The value of this field
    +   * can be used as the contents of the corresponding RDD generated from the physical plan of this
    +   * command.
    +   *
    +   * The `execute()` method of all the physical command classes should reference `sideEffectResult`
    +   * so that the command can be executed eagerly right after the command query is created.
    +   */
    +  protected[sql] lazy val sideEffectResult: Seq[InternalRow] = {
    +    val converter = CatalystTypeConverters.createToCatalystConverter(schema)
    +    val rows = cmd.run(sqlContext.sparkSession, children)
    +
    +    rows.map(converter(_).asInstanceOf[InternalRow])
    +  }
    +
    +  override def innerChildren: Seq[QueryPlan[_]] = cmd.children
    --- End diff --
    
    why do we need this?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85313 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85313/testReport)** for PR 20020 at commit [`b60f4ec`](https://github.com/apache/spark/commit/b60f4ec187799214ba72a8b0ff8bec484f0818a1).
     * This patch **fails to build**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    > since now we let inserting commands have a child, it makes sense to wrap the child with AnalysisBarrier, to avoid re-analyzing them.
    
    It makes sense to me too.
    



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/20020


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85249 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85249/testReport)** for PR 20020 at commit [`83e0fba`](https://github.com/apache/spark/commit/83e0fba0574cf41717f6d9557df17587e27b5301).
     * This patch **fails PySpark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85272 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85272/testReport)** for PR 20020 at commit [`f76bef7`](https://github.com/apache/spark/commit/f76bef78658054c160d18611029c6c3f283876af).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85113 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85113/testReport)** for PR 20020 at commit [`2d58187`](https://github.com/apache/spark/commit/2d58187ac7993e90fb43a35cae4bbd4b0b01a218).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85397 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85397/testReport)** for PR 20020 at commit [`cd2bbf8`](https://github.com/apache/spark/commit/cd2bbf8434a6b142f89f427db8654aeef36cec11).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158199084
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/DataWritingCommand.scala ---
    @@ -20,30 +20,32 @@ package org.apache.spark.sql.execution.command
     import org.apache.hadoop.conf.Configuration
     
     import org.apache.spark.SparkContext
    -import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.{Row, SparkSession}
    +import org.apache.spark.sql.catalyst.expressions.Attribute
    +import org.apache.spark.sql.catalyst.plans.logical.{AnalysisBarrier, Command, LogicalPlan}
    +import org.apache.spark.sql.execution.SparkPlan
     import org.apache.spark.sql.execution.datasources.BasicWriteJobStatsTracker
    +import org.apache.spark.sql.execution.datasources.FileFormatWriter
     import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
     import org.apache.spark.util.SerializableConfiguration
     
    -
     /**
      * A special `RunnableCommand` which writes data out and updates metrics.
      */
    -trait DataWritingCommand extends RunnableCommand {
    -
    +trait DataWritingCommand extends Command {
       /**
        * The input query plan that produces the data to be written.
    +   * IMPORTANT: the input query plan MUST be analyzed, so that we can carry its output columns
    +   *            to [[FileFormatWriter]].
        */
       def query: LogicalPlan
     
    -  // We make the input `query` an inner child instead of a child in order to hide it from the
    -  // optimizer. This is because optimizer may not preserve the output schema names' case, and we
    -  // have to keep the original analyzed plan here so that we can pass the corrected schema to the
    -  // writer. The schema of analyzed plan is what user expects(or specifies), so we should respect
    -  // it when writing.
    -  override protected def innerChildren: Seq[LogicalPlan] = query :: Nil
    +  override def children: Seq[LogicalPlan] = query :: Nil
    --- End diff --
    
    Add `AnalysisBarrier` around `query` here?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158290491
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/DataWritingCommand.scala ---
    @@ -57,4 +59,6 @@ trait DataWritingCommand extends RunnableCommand {
         val serializableHadoopConf = new SerializableConfiguration(hadoopConf)
         new BasicWriteJobStatsTracker(serializableHadoopConf, metrics)
       }
    +
    +  def run(sparkSession: SparkSession, children: Seq[SparkPlan]): Seq[Row]
    --- End diff --
    
    since we define children as `query :: Nil`, here we can just pass `query: SparkPlan`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85313/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85266/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158219553
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala ---
    @@ -87,6 +87,51 @@ case class ExecutedCommandExec(cmd: RunnableCommand) extends LeafExecNode {
       }
     }
    --- End diff --
    
    Well, I prefer to keep it. It is used only in `ExecutedCommandExec`, and in the future if there are new `RunnableCommand` with metrics, the new command can just override `metrics`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85314 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85314/testReport)** for PR 20020 at commit [`787e677`](https://github.com/apache/spark/commit/787e6775ec2ccbcfdcd88e9460f14e4d7f658a98).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158329686
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala ---
    @@ -87,6 +87,44 @@ case class ExecutedCommandExec(cmd: RunnableCommand) extends LeafExecNode {
       }
     }
     
    +/**
    + * A physical operator that executes the run method of a `DataWritingCommand` and
    + * saves the result to prevent multiple executions.
    + *
    + * @param cmd the `DataWritingCommand` this operator will run.
    + * @param child the physical plan child ran by the `DataWritingCommand`.
    + */
    +case class DataWritingCommandExec(cmd: DataWritingCommand, child: SparkPlan)
    +  extends SparkPlan {
    +
    +  override lazy val metrics: Map[String, SQLMetric] = cmd.metrics
    +
    +  protected[sql] lazy val sideEffectResult: Seq[InternalRow] = {
    +    val converter = CatalystTypeConverters.createToCatalystConverter(schema)
    +    val rows = cmd.run(sqlContext.sparkSession, child)
    +
    +    rows.map(converter(_).asInstanceOf[InternalRow])
    +  }
    +
    +  override def children: Seq[SparkPlan] = child :: Nil
    +
    +  override def innerChildren: Seq[QueryPlan[_]] = cmd :: Nil
    --- End diff --
    
    Do we need this?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85269/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85272/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85269 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85269/testReport)** for PR 20020 at commit [`5818c32`](https://github.com/apache/spark/commit/5818c329f043b756d792d01b084469577b8c8120).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158197728
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/DataWritingCommand.scala ---
    @@ -20,30 +20,32 @@ package org.apache.spark.sql.execution.command
     import org.apache.hadoop.conf.Configuration
     
     import org.apache.spark.SparkContext
    -import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.{Row, SparkSession}
    +import org.apache.spark.sql.catalyst.expressions.Attribute
    +import org.apache.spark.sql.catalyst.plans.logical.{AnalysisBarrier, Command, LogicalPlan}
    +import org.apache.spark.sql.execution.SparkPlan
     import org.apache.spark.sql.execution.datasources.BasicWriteJobStatsTracker
    +import org.apache.spark.sql.execution.datasources.FileFormatWriter
     import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
     import org.apache.spark.util.SerializableConfiguration
     
    -
     /**
      * A special `RunnableCommand` which writes data out and updates metrics.
    --- End diff --
    
    Not a `RunnableCommand` now.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    A late LGTM!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158290698
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala ---
    @@ -118,13 +118,14 @@ class FileStreamSink(
               throw new RuntimeException(s"Partition column $col not found in schema ${data.schema}")
             }
           }
    +      val plan = data.queryExecution.executedPlan
     
           FileFormatWriter.write(
             sparkSession = sparkSession,
    -        queryExecution = data.queryExecution,
    +        plan = plan,
             fileFormat = fileFormat,
             committer = committer,
    -        outputSpec = FileFormatWriter.OutputSpec(path, Map.empty),
    +        outputSpec = FileFormatWriter.OutputSpec(path, Map.empty, plan.output),
    --- End diff --
    
    we need logical plan output not physical plan output


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85268 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85268/testReport)** for PR 20020 at commit [`30535b6`](https://github.com/apache/spark/commit/30535b61c1f5c7443d1a5341086d71578ec5950f).
     * This patch **fails to build**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `case class DataWritingCommandExec(cmd: DataWritingCommand, child: SparkPlan)`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85268 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85268/testReport)** for PR 20020 at commit [`30535b6`](https://github.com/apache/spark/commit/30535b61c1f5c7443d1a5341086d71578ec5950f).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85110/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85249/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    @gatorsmile This PR focus on the commands that using `FileFormatWriter`. I will investigate how to resolve the other Insertion commands and create a followup PR, is that OK?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158198088
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala ---
    @@ -87,6 +87,51 @@ case class ExecutedCommandExec(cmd: RunnableCommand) extends LeafExecNode {
       }
     }
     
    +/**
    + * A physical operator that executes the run method of a `RunnableCommand` and
    + * saves the result to prevent multiple executions.
    + *
    + * @param cmd the `RunnableCommand` this operator will run.
    --- End diff --
    
    `DataWritingCommand` is not a `RunnableCommand`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158288419
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala ---
    @@ -87,6 +87,51 @@ case class ExecutedCommandExec(cmd: RunnableCommand) extends LeafExecNode {
       }
     }
     
    +/**
    + * A physical operator that executes the run method of a `DataWritingCommand` and
    + * saves the result to prevent multiple executions.
    + *
    + * @param cmd the `DataWritingCommand` this operator will run.
    + * @param children the children physical plans ran by the `DataWritingCommand`.
    + */
    +case class DataWritingCommandExec(cmd: DataWritingCommand, children: Seq[SparkPlan])
    +  extends SparkPlan {
    +
    +  override lazy val metrics: Map[String, SQLMetric] = cmd.metrics
    +
    +  /**
    +   * A concrete command should override this lazy field to wrap up any side effects caused by the
    --- End diff --
    
    is this comment valid?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    How about the command `SaveIntoDataSourceCommand`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85110 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85110/testReport)** for PR 20020 at commit [`d354895`](https://github.com/apache/spark/commit/d354895031ba42c28ee0d8665f67ccc6e3766821).
     * This patch **fails to build**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `trait DataWritingCommand extends LogicalPlan `
      * `case class DataWritingCommandExec(cmd: DataWritingCommand, children: Seq[SparkPlan])`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158344489
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
    @@ -479,14 +468,15 @@ case class DataSource(
           outputPath = outputPath,
           staticPartitions = Map.empty,
           ifPartitionNotExists = false,
    -      partitionColumns = partitionAttributes,
    +      partitionColumns = partitionColumns.map(UnresolvedAttribute.quoted),
           bucketSpec = bucketSpec,
           fileFormat = format,
           options = options,
           query = data,
    --- End diff --
    
    How about adding an `AnalysisBarrier` here?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85159/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158198595
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/DataWritingCommand.scala ---
    @@ -20,30 +20,32 @@ package org.apache.spark.sql.execution.command
     import org.apache.hadoop.conf.Configuration
     
     import org.apache.spark.SparkContext
    -import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.{Row, SparkSession}
    +import org.apache.spark.sql.catalyst.expressions.Attribute
    +import org.apache.spark.sql.catalyst.plans.logical.{AnalysisBarrier, Command, LogicalPlan}
    +import org.apache.spark.sql.execution.SparkPlan
     import org.apache.spark.sql.execution.datasources.BasicWriteJobStatsTracker
    +import org.apache.spark.sql.execution.datasources.FileFormatWriter
     import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
     import org.apache.spark.util.SerializableConfiguration
     
    -
     /**
      * A special `RunnableCommand` which writes data out and updates metrics.
      */
    -trait DataWritingCommand extends RunnableCommand {
    -
    +trait DataWritingCommand extends Command {
       /**
        * The input query plan that produces the data to be written.
    +   * IMPORTANT: the input query plan MUST be analyzed, so that we can carry its output columns
    +   *            to [[FileFormatWriter]].
        */
       def query: LogicalPlan
     
    -  // We make the input `query` an inner child instead of a child in order to hide it from the
    -  // optimizer. This is because optimizer may not preserve the output schema names' case, and we
    -  // have to keep the original analyzed plan here so that we can pass the corrected schema to the
    -  // writer. The schema of analyzed plan is what user expects(or specifies), so we should respect
    -  // it when writing.
    -  override protected def innerChildren: Seq[LogicalPlan] = query :: Nil
    +  override def children: Seq[LogicalPlan] = query :: Nil
     
    -  override lazy val metrics: Map[String, SQLMetric] = {
    +  // Output columns of the analyzed input query plan
    +  def allColumns: Seq[Attribute]
    --- End diff --
    
    `outputColumns`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85161/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158200471
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/DataWritingCommand.scala ---
    @@ -20,30 +20,32 @@ package org.apache.spark.sql.execution.command
     import org.apache.hadoop.conf.Configuration
     
     import org.apache.spark.SparkContext
    -import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.{Row, SparkSession}
    +import org.apache.spark.sql.catalyst.expressions.Attribute
    +import org.apache.spark.sql.catalyst.plans.logical.{AnalysisBarrier, Command, LogicalPlan}
    +import org.apache.spark.sql.execution.SparkPlan
     import org.apache.spark.sql.execution.datasources.BasicWriteJobStatsTracker
    +import org.apache.spark.sql.execution.datasources.FileFormatWriter
     import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
     import org.apache.spark.util.SerializableConfiguration
     
    -
     /**
      * A special `RunnableCommand` which writes data out and updates metrics.
      */
    -trait DataWritingCommand extends RunnableCommand {
    -
    +trait DataWritingCommand extends Command {
       /**
        * The input query plan that produces the data to be written.
    +   * IMPORTANT: the input query plan MUST be analyzed, so that we can carry its output columns
    +   *            to [[FileFormatWriter]].
        */
       def query: LogicalPlan
     
    -  // We make the input `query` an inner child instead of a child in order to hide it from the
    -  // optimizer. This is because optimizer may not preserve the output schema names' case, and we
    -  // have to keep the original analyzed plan here so that we can pass the corrected schema to the
    -  // writer. The schema of analyzed plan is what user expects(or specifies), so we should respect
    -  // it when writing.
    -  override protected def innerChildren: Seq[LogicalPlan] = query :: Nil
    +  override def children: Seq[LogicalPlan] = query :: Nil
    --- End diff --
    
    I tried but it will cause runtime exception. When the `AnalysisBarrier` is removed by analyzer,  the child of `DataWritingCommand` is no longer `AnalysisBarrier` 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85313 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85313/testReport)** for PR 20020 at commit [`b60f4ec`](https://github.com/apache/spark/commit/b60f4ec187799214ba72a8b0ff8bec484f0818a1).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85113/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85269 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85269/testReport)** for PR 20020 at commit [`5818c32`](https://github.com/apache/spark/commit/5818c329f043b756d792d01b084469577b8c8120).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r212850054
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/DataWritingCommand.scala ---
    @@ -20,30 +20,32 @@ package org.apache.spark.sql.execution.command
     import org.apache.hadoop.conf.Configuration
     
     import org.apache.spark.SparkContext
    -import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.{Row, SparkSession}
    +import org.apache.spark.sql.catalyst.expressions.Attribute
    +import org.apache.spark.sql.catalyst.plans.logical.{Command, LogicalPlan}
    +import org.apache.spark.sql.execution.SparkPlan
     import org.apache.spark.sql.execution.datasources.BasicWriteJobStatsTracker
    +import org.apache.spark.sql.execution.datasources.FileFormatWriter
     import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
     import org.apache.spark.util.SerializableConfiguration
     
    -
     /**
    - * A special `RunnableCommand` which writes data out and updates metrics.
    + * A special `Command` which writes data out and updates metrics.
      */
    -trait DataWritingCommand extends RunnableCommand {
    -
    +trait DataWritingCommand extends Command {
       /**
        * The input query plan that produces the data to be written.
    +   * IMPORTANT: the input query plan MUST be analyzed, so that we can carry its output columns
    +   *            to [[FileFormatWriter]].
        */
       def query: LogicalPlan
     
    -  // We make the input `query` an inner child instead of a child in order to hide it from the
    -  // optimizer. This is because optimizer may not preserve the output schema names' case, and we
    -  // have to keep the original analyzed plan here so that we can pass the corrected schema to the
    -  // writer. The schema of analyzed plan is what user expects(or specifies), so we should respect
    -  // it when writing.
    -  override protected def innerChildren: Seq[LogicalPlan] = query :: Nil
    +  override final def children: Seq[LogicalPlan] = query :: Nil
     
    -  override lazy val metrics: Map[String, SQLMetric] = {
    +  // Output columns of the analyzed input query plan
    +  def outputColumns: Seq[Attribute]
    --- End diff --
    
    `outputColumns` changed from analyzed to optimized. For example:
    ```scala
    withTempDir { dir =>
      val path = dir.getCanonicalPath
      val cnt = 30
      val table1Path = s"$path/table1"
      val table3Path = s"$path/table3"
      spark.range(cnt).selectExpr("cast(id as bigint) as col1", "cast(id % 3 as bigint) as col2")
        .write.mode(SaveMode.Overwrite).parquet(table1Path)
      withTable("table1", "table3") {
        spark.sql(
          s"CREATE TABLE table1(col1 bigint, col2 bigint) using parquet location '$table1Path/'")
        spark.sql("CREATE TABLE table3(COL1 bigint, COL2 bigint) using parquet " +
          "PARTITIONED BY (COL2) " +
          s"CLUSTERED BY (COL1) INTO 2 BUCKETS location '$table3Path/'")
    
        withView("view1") {
          spark.sql("CREATE VIEW view1 as select col1, col2 from table1 where col1 > -20")
          spark.sql("INSERT OVERWRITE TABLE table3 select COL1, COL2 from view1 CLUSTER BY COL1")
          spark.table("table3").show
        }
      }
    }
    ```
    ```
    outputColumns: List(COL1#19L, COL2#20L)
    outputColumns: List(COL1#19L, COL2#20L)
    outputColumns: List(COL1#19L, COL2#20L)
    outputColumns: List(COL1#19L, COL2#20L)
    outputColumns: List(COL1#19L, COL2#20L)
    outputColumns: List(COL1#19L, COL2#20L)
    outputColumns: List(COL1#19L, COL2#20L)
    outputColumns: List(COL1#19L, COL2#20L)
    outputColumns: List(col1#16L, col2#17L)
    outputColumns: List(col1#16L, col2#17L)
    outputColumns: List(col1#16L, col2#17L)
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158018917
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/DataWritingCommand.scala ---
    @@ -57,4 +45,8 @@ trait DataWritingCommand extends RunnableCommand {
         val serializableHadoopConf = new SerializableConfiguration(hadoopConf)
         new BasicWriteJobStatsTracker(serializableHadoopConf, metrics)
       }
    +
    +  def run(sparkSession: SparkSession, children: Seq[SparkPlan]): Seq[Row] = {
    +    throw new NotImplementedError
    --- End diff --
    
    shall we force all sub-classes to implement it?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85314/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85193 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85193/testReport)** for PR 20020 at commit [`7ccfd90`](https://github.com/apache/spark/commit/7ccfd909a71a531be25ab0eccdf7b347d06c2c0a).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158198250
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala ---
    @@ -87,6 +87,51 @@ case class ExecutedCommandExec(cmd: RunnableCommand) extends LeafExecNode {
       }
     }
    --- End diff --
    
    I think the `metrics` in `RunnableCommand` is mainly for use in `DataWritingCommand`. Now `DataWritingCommand` is a separate class other than `RunnableCommand`, do we still need `metrics` in `RunnableCommand`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    All these insertion commands are from `postHocResolutionRules`, while there are other batches after it.  Skipping the batches after `postHocResolutionRules` will cause analysis error.
    I decide not to add `AnalysisBarrier` for correctness and robustness.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85225 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85225/testReport)** for PR 20020 at commit [`7ccfd90`](https://github.com/apache/spark/commit/7ccfd909a71a531be25ab0eccdf7b347d06c2c0a).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    thanks, merging to master!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85161 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85161/testReport)** for PR 20020 at commit [`e25a9eb`](https://github.com/apache/spark/commit/e25a9eb285d56a771a56b77534413be59b9f111b).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158341954
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/DataWritingCommand.scala ---
    @@ -20,30 +20,32 @@ package org.apache.spark.sql.execution.command
     import org.apache.hadoop.conf.Configuration
     
     import org.apache.spark.SparkContext
    -import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.{Row, SparkSession}
    +import org.apache.spark.sql.catalyst.expressions.Attribute
    +import org.apache.spark.sql.catalyst.plans.logical.{AnalysisBarrier, Command, LogicalPlan}
    --- End diff --
    
    `AnalysisBarrier ` is not needed any more.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85193 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85193/testReport)** for PR 20020 at commit [`7ccfd90`](https://github.com/apache/spark/commit/7ccfd909a71a531be25ab0eccdf7b347d06c2c0a).
     * This patch **fails PySpark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85110 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85110/testReport)** for PR 20020 at commit [`d354895`](https://github.com/apache/spark/commit/d354895031ba42c28ee0d8665f67ccc6e3766821).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158344919
  
    --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala ---
    @@ -148,7 +148,8 @@ object HiveAnalysis extends Rule[LogicalPlan] {
       override def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
         case InsertIntoTable(r: HiveTableRelation, partSpec, query, overwrite, ifPartitionNotExists)
             if DDLUtils.isHiveTable(r.tableMeta) =>
    -      InsertIntoHiveTable(r.tableMeta, partSpec, query, overwrite, ifPartitionNotExists)
    +      InsertIntoHiveTable(r.tableMeta, partSpec, query, overwrite,
    --- End diff --
    
    Add a `AnalysisBarrier ` here?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85314 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85314/testReport)** for PR 20020 at commit [`787e677`](https://github.com/apache/spark/commit/787e6775ec2ccbcfdcd88e9460f14e4d7f658a98).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85493/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158333591
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala ---
    @@ -87,6 +87,44 @@ case class ExecutedCommandExec(cmd: RunnableCommand) extends LeafExecNode {
       }
     }
     
    +/**
    + * A physical operator that executes the run method of a `DataWritingCommand` and
    + * saves the result to prevent multiple executions.
    + *
    + * @param cmd the `DataWritingCommand` this operator will run.
    + * @param child the physical plan child ran by the `DataWritingCommand`.
    + */
    +case class DataWritingCommandExec(cmd: DataWritingCommand, child: SparkPlan)
    +  extends SparkPlan {
    +
    +  override lazy val metrics: Map[String, SQLMetric] = cmd.metrics
    +
    +  protected[sql] lazy val sideEffectResult: Seq[InternalRow] = {
    +    val converter = CatalystTypeConverters.createToCatalystConverter(schema)
    +    val rows = cmd.run(sqlContext.sparkSession, child)
    +
    +    rows.map(converter(_).asInstanceOf[InternalRow])
    +  }
    +
    +  override def children: Seq[SparkPlan] = child :: Nil
    +
    +  override def innerChildren: Seq[QueryPlan[_]] = cmd :: Nil
    --- End diff --
    
    I was following `ExecutedCommandExec`. And I can see some slight difference with`explain` command. I can remove it.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85193/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85249 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85249/testReport)** for PR 20020 at commit [`83e0fba`](https://github.com/apache/spark/commit/83e0fba0574cf41717f6d9557df17587e27b5301).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158882294
  
    --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala ---
    @@ -1970,6 +1970,19 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
         }
       }
     
    +  test("SPARK-22252: FileFormatWriter should respect the input query schema in HIVE") {
    --- End diff --
    
    move it to `HiveCatalogedDDLSuite`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85159/testReport)** for PR 20020 at commit [`e25a9eb`](https://github.com/apache/spark/commit/e25a9eb285d56a771a56b77534413be59b9f111b).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85113 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85113/testReport)** for PR 20020 at commit [`2d58187`](https://github.com/apache/spark/commit/2d58187ac7993e90fb43a35cae4bbd4b0b01a218).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158203446
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/DataWritingCommand.scala ---
    @@ -20,30 +20,32 @@ package org.apache.spark.sql.execution.command
     import org.apache.hadoop.conf.Configuration
     
     import org.apache.spark.SparkContext
    -import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
    +import org.apache.spark.sql.{Row, SparkSession}
    +import org.apache.spark.sql.catalyst.expressions.Attribute
    +import org.apache.spark.sql.catalyst.plans.logical.{AnalysisBarrier, Command, LogicalPlan}
    +import org.apache.spark.sql.execution.SparkPlan
     import org.apache.spark.sql.execution.datasources.BasicWriteJobStatsTracker
    +import org.apache.spark.sql.execution.datasources.FileFormatWriter
     import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
     import org.apache.spark.util.SerializableConfiguration
     
    -
     /**
      * A special `RunnableCommand` which writes data out and updates metrics.
      */
    -trait DataWritingCommand extends RunnableCommand {
    -
    +trait DataWritingCommand extends Command {
       /**
        * The input query plan that produces the data to be written.
    +   * IMPORTANT: the input query plan MUST be analyzed, so that we can carry its output columns
    +   *            to [[FileFormatWriter]].
        */
       def query: LogicalPlan
     
    -  // We make the input `query` an inner child instead of a child in order to hide it from the
    -  // optimizer. This is because optimizer may not preserve the output schema names' case, and we
    -  // have to keep the original analyzed plan here so that we can pass the corrected schema to the
    -  // writer. The schema of analyzed plan is what user expects(or specifies), so we should respect
    -  // it when writing.
    -  override protected def innerChildren: Seq[LogicalPlan] = query :: Nil
    +  override def children: Seq[LogicalPlan] = query :: Nil
    --- End diff --
    
    ah, right. We should add the barrier when passing in the query.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    since now we let inserting commands have a child, it makes sense to wrap the child with `AnalysisBarrier`, to avoid re-analyzing them. also cc @viirya 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85397/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158416541
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ---
    @@ -479,14 +468,15 @@ case class DataSource(
           outputPath = outputPath,
           staticPartitions = Map.empty,
           ifPartitionNotExists = false,
    -      partitionColumns = partitionAttributes,
    +      partitionColumns = partitionColumns.map(UnresolvedAttribute.quoted),
           bucketSpec = bucketSpec,
           fileFormat = format,
           options = options,
           query = data,
    --- End diff --
    
    I think @cloud-fan wants `AnalysisBarrier` to be in trait `DataWritingCommand` to make sure the `query` is analyzed.
    Adding `AnalysisBarrier` will not be helpful, the query should be analyzed here.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20020: [SPARK-22834][SQL] Make insertion commands have r...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20020#discussion_r158341832
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
    @@ -27,7 +27,7 @@ import org.apache.spark.sql.catalyst.plans._
     import org.apache.spark.sql.catalyst.plans.logical._
     import org.apache.spark.sql.catalyst.plans.physical._
     import org.apache.spark.sql.execution.columnar.{InMemoryRelation, InMemoryTableScanExec}
    -import org.apache.spark.sql.execution.command._
    +import org.apache.spark.sql.execution.command.{DataWritingCommand, _}
    --- End diff --
    
    Do we still need this?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85161 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85161/testReport)** for PR 20020 at commit [`e25a9eb`](https://github.com/apache/spark/commit/e25a9eb285d56a771a56b77534413be59b9f111b).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `trait DataWritingCommand extends Command `
      * `case class DataWritingCommandExec(cmd: DataWritingCommand, children: Seq[SparkPlan])`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85397 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85397/testReport)** for PR 20020 at commit [`cd2bbf8`](https://github.com/apache/spark/commit/cd2bbf8434a6b142f89f427db8654aeef36cec11).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85493 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85493/testReport)** for PR 20020 at commit [`18ec016`](https://github.com/apache/spark/commit/18ec01638b9da7f8150e3ea35c2876d6d1f41f3d).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    **[Test build #85396 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85396/testReport)** for PR 20020 at commit [`cd2bbf8`](https://github.com/apache/spark/commit/cd2bbf8434a6b142f89f427db8654aeef36cec11).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85225/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20020: [SPARK-22834][SQL] Make insertion commands have real chi...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20020
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org