You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by ericl <gi...@git.apache.org> on 2018/04/04 00:30:55 UTC

[GitHub] spark pull request #20971: [SPARK-23809][SQL][backport] Active SparkSession ...

GitHub user ericl opened a pull request:

    https://github.com/apache/spark/pull/20971

    [SPARK-23809][SQL][backport] Active SparkSession should be set by getOrCreate

    This backports https://github.com/apache/spark/pull/20927 to branch-2.3
    
    ## What changes were proposed in this pull request?
    
    Currently, the active spark session is set inconsistently (e.g., in createDataFrame, prior to query execution). Many places in spark also incorrectly query active session when they should be calling activeSession.getOrElse(defaultSession) and so might get None even if a Spark session exists.
    
    The semantics here can be cleaned up if we also set the active session when the default session is set.
    
    Related: https://github.com/apache/spark/pull/20926/files
    
    ## How was this patch tested?
    
    Unit test, existing test. Note that if https://github.com/apache/spark/pull/20926 merges first we should also update the tests there.
    
    Author: Eric Liang <ek...@databricks.com>
    
    Closes #20927 from ericl/active-session-cleanup.
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ericl/spark backport-23809

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20971.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20971
    
----
commit f2303dcef61660dabfd08be5568b7da10cf1b117
Author: Eric Liang <ek...@...>
Date:   2018-04-04T00:09:12Z

    [SPARK-23809][SQL] Active SparkSession should be set by getOrCreate
    
    ## What changes were proposed in this pull request?
    
    Currently, the active spark session is set inconsistently (e.g., in createDataFrame, prior to query execution). Many places in spark also incorrectly query active session when they should be calling activeSession.getOrElse(defaultSession) and so might get None even if a Spark session exists.
    
    The semantics here can be cleaned up if we also set the active session when the default session is set.
    
    Related: https://github.com/apache/spark/pull/20926/files
    
    ## How was this patch tested?
    
    Unit test, existing test. Note that if https://github.com/apache/spark/pull/20926 merges first we should also update the tests there.
    
    Author: Eric Liang <ek...@databricks.com>
    
    Closes #20927 from ericl/active-session-cleanup.

commit e429af1e9a5a2f8ed3e90ee215d561c05aeb33b3
Author: Eric Liang <ek...@...>
Date:   2018-04-04T00:30:50Z

    Tue Apr  3 17:30:50 PDT 2018

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20971: [SPARK-23809][SQL][backport] Active SparkSession should ...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20971
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88865/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20971: [SPARK-23809][SQL][backport] Active SparkSession should ...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20971
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20971: [SPARK-23809][SQL][backport] Active SparkSession should ...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20971
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20971: [SPARK-23809][SQL][backport] Active SparkSession should ...

Posted by gatorsmile <gi...@git.apache.org>.

Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/20971
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20971: [SPARK-23809][SQL][backport] Active SparkSession should ...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20971
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88961/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20971: [SPARK-23809][SQL][backport] Active SparkSession should ...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20971
  
    **[Test build #88870 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88870/testReport)** for PR 20971 at commit [`36fa1bd`](https://github.com/apache/spark/commit/36fa1bdc847f0b5ffb61284a35f3183751255705).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20971: [SPARK-23809][SQL][backport] Active SparkSession should ...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20971
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88867/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20971: [SPARK-23809][SQL][backport] Active SparkSession should ...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20971
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20971: [SPARK-23809][SQL][backport] Active SparkSession should ...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20971
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org