You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tdas <gi...@git.apache.org> on 2017/09/15 00:34:27 UTC

[GitHub] spark pull request #19240: [SPARK-22018][SQL]Preserve top-level alias metada...

GitHub user tdas opened a pull request:

    https://github.com/apache/spark/pull/19240

    [SPARK-22018][SQL]Preserve top-level alias metadata when collapsing projects

    ## What changes were proposed in this pull request?
    If there are two projects like as follows.
    ```
    Project [a_with_metadata#27 AS b#26]
    +- Project [a#0 AS a_with_metadata#27]
       +- LocalRelation <empty>, [a#0, b#1]
    ```
    Child Project has an output column with a metadata in it, and the parent Project has an alias that implicitly forwards the metadata. So this metadata is visible for higher operators. Upon applying CollapseProject optimizer rule, the metadata is not preserved.
    ```
    Project [a#0 AS b#26]
    +- LocalRelation <empty>, [a#0, b#1]
    ```
    This is incorrect, as downstream operators that expect certain metadata (e.g. watermark in structured streaming) to identify certain fields will fail to do so. This PR fixes it by preserving the metadata of top-level aliases.
    
    ## How was this patch tested?
    New unit test


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tdas/spark SPARK-22018

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19240.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19240
    
----
commit b3e41a7603e8bf917e9b596bdeb6afa51a32a695
Author: Tathagata Das <ta...@gmail.com>
Date:   2017-09-15T00:29:56Z

    Peserver top-level alias metadata

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19240: [SPARK-22018][SQL]Preserve top-level alias metadata when...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19240
  
    **[Test build #81801 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81801/testReport)** for PR 19240 at commit [`b3e41a7`](https://github.com/apache/spark/commit/b3e41a7603e8bf917e9b596bdeb6afa51a32a695).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19240: [SPARK-22018][SQL]Preserve top-level alias metadata when...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19240
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81801/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19240: [SPARK-22018][SQL]Preserve top-level alias metadata when...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19240
  
    **[Test build #81801 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81801/testReport)** for PR 19240 at commit [`b3e41a7`](https://github.com/apache/spark/commit/b3e41a7603e8bf917e9b596bdeb6afa51a32a695).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19240: [SPARK-22018][SQL]Preserve top-level alias metadata when...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the issue:

    https://github.com/apache/spark/pull/19240
  
    LGTM


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19240: [SPARK-22018][SQL]Preserve top-level alias metada...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/19240


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19240: [SPARK-22018][SQL]Preserve top-level alias metada...

Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19240#discussion_r139042816
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
    @@ -2256,7 +2256,10 @@ object CleanupAliases extends Rule[LogicalPlan] {
     
       def trimNonTopLevelAliases(e: Expression): Expression = e match {
         case a: Alias =>
    -      a.withNewChildren(trimAliases(a.child) :: Nil)
    --- End diff --
    
    Is there anywhere else in the code base where we call `withNewChildren` on an alias?  Should we just override that?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19240: [SPARK-22018][SQL]Preserve top-level alias metadata when...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19240
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org