You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by zjffdu <gi...@git.apache.org> on 2016/09/23 05:13:56 UTC

[GitHub] zeppelin pull request #1452: ZEPPELIN-1442. UDF can not be found due to 2 in...

GitHub user zjffdu opened a pull request:

    https://github.com/apache/zeppelin/pull/1452

    ZEPPELIN-1442. UDF can not be found due to 2 instances of SparkSession is created

    ### What is this PR for?
    The issue is that we create 2 SparkSession in zeppelin_pyspark.py (Because we create SQLContext first which will create SparkSession underlying). This cause 2 instances of SparkSession in JVM side and this means we have 2 instances of Catalog as well. So udf registered in SQLContext can be used in SparkSession. This PR will create SparkSession first and then assign its internal SQLContext to sqlContext in pyspark. 
    
    
    ### What type of PR is it?
    [Bug Fix]
    
    ### Todos
    * [ ] - Task
    
    ### What is the Jira issue?
    * https://issues.apache.org/jira/browse/ZEPPELIN-1442
    
    ### How should this be tested?
    Integration test is added. 
    
    ### Screenshots (if appropriate)
    ![image](https://cloud.githubusercontent.com/assets/164491/18774832/7f270de4-818f-11e6-9e4f-c4def4353e5c.png)
    
    ### Questions:
    * Does the licenses files need update? No
    * Is there breaking changes for older versions? No
    * Does this needs documentation? No
    
    \u2026

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1442

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/zeppelin/pull/1452.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1452
    
----
commit 948e8657634686f16a405b3938884a5fe48dfc1c
Author: Jeff Zhang <zj...@apache.org>
Date:   2016-09-23T05:08:49Z

    ZEPPELIN-1442. UDF can not be found due to 2 instances of SparkSession is created

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1452: ZEPPELIN-1442. UDF can not be found due to 2 in...

Posted by zjffdu <gi...@git.apache.org>.
GitHub user zjffdu reopened a pull request:

    https://github.com/apache/zeppelin/pull/1452

    ZEPPELIN-1442. UDF can not be found due to 2 instances of SparkSession is created

    ### What is this PR for?
    The issue is that we create 2 SparkSession in zeppelin_pyspark.py (Because we create SQLContext first which will create SparkSession underlying). This cause 2 instances of SparkSession in JVM side and this means we have 2 instances of Catalog as well. So udf registered in SQLContext can be used in SparkSession. This PR will create SparkSession first and then assign its internal SQLContext to sqlContext in pyspark. 
    
    
    ### What type of PR is it?
    [Bug Fix]
    
    ### Todos
    * [ ] - Task
    
    ### What is the Jira issue?
    * https://issues.apache.org/jira/browse/ZEPPELIN-1442
    
    ### How should this be tested?
    Integration test is added. 
    
    ### Screenshots (if appropriate)
    ![image](https://cloud.githubusercontent.com/assets/164491/18774832/7f270de4-818f-11e6-9e4f-c4def4353e5c.png)
    
    ### Questions:
    * Does the licenses files need update? No
    * Is there breaking changes for older versions? No
    * Does this needs documentation? No
    
    \u2026

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1442

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/zeppelin/pull/1452.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1452
    
----
commit 948e8657634686f16a405b3938884a5fe48dfc1c
Author: Jeff Zhang <zj...@apache.org>
Date:   2016-09-23T05:08:49Z

    ZEPPELIN-1442. UDF can not be found due to 2 instances of SparkSession is created

commit 66ec1b9cf5f244ec3b61594f3de86c34ca117175
Author: Jeff Zhang <zj...@apache.org>
Date:   2016-09-23T08:18:24Z

    fix unit test

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1452: ZEPPELIN-1442. UDF can not be found due to 2 in...

Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu closed the pull request at:

    https://github.com/apache/zeppelin/pull/1452


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1452: ZEPPELIN-1442. UDF can not be found due to 2 in...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/zeppelin/pull/1452


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1452: ZEPPELIN-1442. UDF can not be found due to 2 instances...

Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:

    https://github.com/apache/zeppelin/pull/1452
  
    LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1452: ZEPPELIN-1442. UDF can not be found due to 2 instances...

Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:

    https://github.com/apache/zeppelin/pull/1452
  
    @minahlee  PR is rebased, and the failed test is irrelevant.
    
    ```
    - should provide onclick method *** FAILED ***
      The code passed to eventually never returned normally. Attempted 1 times over 325.359079 milliseconds. Last failure message: 0 was not equal to 1. (AbstractAngularElemTest.scala:72)
    AngularElem
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1452: ZEPPELIN-1442. UDF can not be found due to 2 instances...

Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:

    https://github.com/apache/zeppelin/pull/1452
  
    LGTM
    @zjffdu Do you mind trigger CI one more time?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1452: ZEPPELIN-1442. UDF can not be found due to 2 instances...

Posted by minahlee <gi...@git.apache.org>.
Github user minahlee commented on the issue:

    https://github.com/apache/zeppelin/pull/1452
  
    @zjffdu Thank you! Merging to master and branch-0.6 if there is no more discussion


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin issue #1452: ZEPPELIN-1442. UDF can not be found due to 2 instances...

Posted by minahlee <gi...@git.apache.org>.
Github user minahlee commented on the issue:

    https://github.com/apache/zeppelin/pull/1452
  
    @zjffdu could you rebase and resolve conflicts?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1452: ZEPPELIN-1442. UDF can not be found due to 2 in...

Posted by zjffdu <gi...@git.apache.org>.
GitHub user zjffdu reopened a pull request:

    https://github.com/apache/zeppelin/pull/1452

    ZEPPELIN-1442. UDF can not be found due to 2 instances of SparkSession is created

    ### What is this PR for?
    The issue is that we create 2 SparkSession in zeppelin_pyspark.py (Because we create SQLContext first which will create SparkSession underlying). This cause 2 instances of SparkSession in JVM side and this means we have 2 instances of Catalog as well. So udf registered in SQLContext can be used in SparkSession. This PR will create SparkSession first and then assign its internal SQLContext to sqlContext in pyspark. 
    
    
    ### What type of PR is it?
    [Bug Fix]
    
    ### Todos
    * [ ] - Task
    
    ### What is the Jira issue?
    * https://issues.apache.org/jira/browse/ZEPPELIN-1442
    
    ### How should this be tested?
    Integration test is added. 
    
    ### Screenshots (if appropriate)
    ![image](https://cloud.githubusercontent.com/assets/164491/18774832/7f270de4-818f-11e6-9e4f-c4def4353e5c.png)
    
    ### Questions:
    * Does the licenses files need update? No
    * Is there breaking changes for older versions? No
    * Does this needs documentation? No
    
    \u2026

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1442

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/zeppelin/pull/1452.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1452
    
----
commit 948e8657634686f16a405b3938884a5fe48dfc1c
Author: Jeff Zhang <zj...@apache.org>
Date:   2016-09-23T05:08:49Z

    ZEPPELIN-1442. UDF can not be found due to 2 instances of SparkSession is created

commit 66ec1b9cf5f244ec3b61594f3de86c34ca117175
Author: Jeff Zhang <zj...@apache.org>
Date:   2016-09-23T08:18:24Z

    fix unit test

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] zeppelin pull request #1452: ZEPPELIN-1442. UDF can not be found due to 2 in...

Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu closed the pull request at:

    https://github.com/apache/zeppelin/pull/1452


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---