You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by sohama4 <gi...@git.apache.org> on 2018/05/13 22:40:47 UTC

[GitHub] spark pull request #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withN...

GitHub user sohama4 opened a pull request:

    https://github.com/apache/spark/pull/21316

    [SPARK-20538][SQL] Wrap Dataset.reduce with withNewExecutionId.

    ## What changes were proposed in this pull request?
    
    Wrap Dataset.reduce with `withNewExecutionId`.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/sohama4/spark dataset_reduce_withexecutionid

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21316.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21316
    
----
commit 9b507bc4312e0cadd84a6436a0311e739ae22d18
Author: Soham Aurangabadkar <so...@...>
Date:   2018-05-13T22:19:05Z

    Wrap Dataset.reduce with withNewExecutionId.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withN...

Posted by jaceklaskowski <gi...@git.apache.org>.
Github user jaceklaskowski commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21316#discussion_r187899299
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
    @@ -1607,7 +1607,9 @@ class Dataset[T] private[sql](
        */
       @Experimental
       @InterfaceStability.Evolving
    -  def reduce(func: (T, T) => T): T = rdd.reduce(func)
    +  def reduce(func: (T, T) => T): T = withNewExecutionId {
    --- End diff --
    
    @maropu When you asked about this API did you refer to `reduce` or `withNewExecutionId`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withNewRddEx...

Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:

    https://github.com/apache/spark/pull/21316
  
    ok to test


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withNewExecu...

Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/21316
  
    Is it better to update title and the PR description, too?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withNewRddEx...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21316
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90718/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withNewRddEx...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21316
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withN...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21316#discussion_r187931208
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
    @@ -1607,7 +1607,9 @@ class Dataset[T] private[sql](
        */
       @Experimental
       @InterfaceStability.Evolving
    -  def reduce(func: (T, T) => T): T = rdd.reduce(func)
    +  def reduce(func: (T, T) => T): T = withNewExecutionId {
    --- End diff --
    
    `reduce`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withN...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21316#discussion_r188104204
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
    @@ -1607,7 +1607,9 @@ class Dataset[T] private[sql](
        */
       @Experimental
       @InterfaceStability.Evolving
    -  def reduce(func: (T, T) => T): T = rdd.reduce(func)
    +  def reduce(func: (T, T) => T): T = withNewExecutionId {
    --- End diff --
    
    Why would we want to deprecate it?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withNewExecu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21316
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withN...

Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21316#discussion_r188107132
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
    @@ -1607,7 +1607,9 @@ class Dataset[T] private[sql](
        */
       @Experimental
       @InterfaceStability.Evolving
    -  def reduce(func: (T, T) => T): T = rdd.reduce(func)
    +  def reduce(func: (T, T) => T): T = withNewExecutionId {
    --- End diff --
    
    this method should use `withNewRDDExecutionId`


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withNewRddEx...

Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:

    https://github.com/apache/spark/pull/21316
  
    Thanks! Merging to master.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withN...

Posted by sohama4 <gi...@git.apache.org>.
Github user sohama4 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21316#discussion_r188143976
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
    @@ -1607,7 +1607,9 @@ class Dataset[T] private[sql](
        */
       @Experimental
       @InterfaceStability.Evolving
    -  def reduce(func: (T, T) => T): T = rdd.reduce(func)
    +  def reduce(func: (T, T) => T): T = withNewExecutionId {
    --- End diff --
    
    Thanks, that makes sense when I looked at the code for `foreach` and `foreachPartition`; I put up a new version with this change. It however wasn't clear immediately how the new function `withNewRDDExecutionId` would be beneficial over `withNewExecutionId`, can you elaborate a little when you get the chance?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withN...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/21316


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withNewExecu...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21316
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withN...

Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21316#discussion_r188871413
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
    @@ -1607,7 +1607,9 @@ class Dataset[T] private[sql](
        */
       @Experimental
       @InterfaceStability.Evolving
    -  def reduce(func: (T, T) => T): T = rdd.reduce(func)
    +  def reduce(func: (T, T) => T): T = withNewExecutionId {
    --- End diff --
    
    @sohama4 Spark Web UI will show no SQL metrics when using `withNewExecutionId`. See https://github.com/apache/spark/pull/19689


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withNewRddEx...

Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:

    https://github.com/apache/spark/pull/21316
  
    LGTM pending tests


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withNewExecu...

Posted by sohama4 <gi...@git.apache.org>.
Github user sohama4 commented on the issue:

    https://github.com/apache/spark/pull/21316
  
    @vanzin @maropu @gatorsmile @jaceklaskowski can someone take a look at this? I'd appreciate it!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withN...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21316#discussion_r187820636
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
    @@ -1607,7 +1607,9 @@ class Dataset[T] private[sql](
        */
       @Experimental
       @InterfaceStability.Evolving
    -  def reduce(func: (T, T) => T): T = rdd.reduce(func)
    +  def reduce(func: (T, T) => T): T = withNewExecutionId {
    --- End diff --
    
    Only a place to need this wrapper in this file?
    
    @gatorsmile btw, this api is still useful? Or, this intends to be deprecated in v3.0+?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withNewRddEx...

Posted by sohama4 <gi...@git.apache.org>.
Github user sohama4 commented on the issue:

    https://github.com/apache/spark/pull/21316
  
    @kiszk @jaceklaskowski @rxin @zsxwing @maropu can we move forward here? Thanks!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withN...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21316#discussion_r187838099
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
    @@ -1607,7 +1607,9 @@ class Dataset[T] private[sql](
        */
       @Experimental
       @InterfaceStability.Evolving
    -  def reduce(func: (T, T) => T): T = rdd.reduce(func)
    +  def reduce(func: (T, T) => T): T = withNewExecutionId {
    --- End diff --
    
    cc @zsxwing 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withNewRddEx...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21316
  
    **[Test build #90718 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90718/testReport)** for PR 21316 at commit [`1f45513`](https://github.com/apache/spark/commit/1f45513c9df0a37faea587be61b92c074642e594).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withN...

Posted by sohama4 <gi...@git.apache.org>.
Github user sohama4 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21316#discussion_r187832727
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
    @@ -1607,7 +1607,9 @@ class Dataset[T] private[sql](
        */
       @Experimental
       @InterfaceStability.Evolving
    -  def reduce(func: (T, T) => T): T = rdd.reduce(func)
    +  def reduce(func: (T, T) => T): T = withNewExecutionId {
    --- End diff --
    
    I believe so.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withNewExecu...

Posted by sohama4 <gi...@git.apache.org>.
Github user sohama4 commented on the issue:

    https://github.com/apache/spark/pull/21316
  
    Thanks for the approval @jaceklaskowski! Can you leave a comment so that Jenkins can get testing underway?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withNewRddEx...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21316
  
    **[Test build #90718 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90718/testReport)** for PR 21316 at commit [`1f45513`](https://github.com/apache/spark/commit/1f45513c9df0a37faea587be61b92c074642e594).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21316: [SPARK-20538][SQL] Wrap Dataset.reduce with withN...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21316#discussion_r189496880
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
    @@ -1607,7 +1607,9 @@ class Dataset[T] private[sql](
        */
       @Experimental
       @InterfaceStability.Evolving
    -  def reduce(func: (T, T) => T): T = rdd.reduce(func)
    +  def reduce(func: (T, T) => T): T = withNewExecutionId {
    --- End diff --
    
    @rxin Sorry, I missed that comment. I just thought what `reduce` can do is what type-safe `groupByKey` can do. But, probably I missed something.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org