You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by holdenk <gi...@git.apache.org> on 2015/10/27 23:51:42 UTC

[GitHub] spark pull request: [SPARK-10658][PYSPARK][WIP] Provide add jars t...

GitHub user holdenk opened a pull request:

    https://github.com/apache/spark/pull/9313

    [SPARK-10658][PYSPARK][WIP] Provide add jars to py spark api

    This does some work to allow dynamic adding of classes to running PySpark instance, although it still suffers from some of the restrictions mentioned in SPARK-5185 but provides a utility method for dealing with that. Should we eventually fix the class loader used by Py4J then this should continue to work and we can kill the helper method.
    
    Something to note for reviewers: I'm using the test JAR also used by R, I could make a copy of it but I figured referencing it would be OK but want to double check.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/holdenk/spark SPARK-10658-provide-addJars-to-pySpark-API

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/9313.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #9313
    
----
commit 5eb089203fb859cebb9564d881e001472d31ffec
Author: Holden Karau <ho...@us.ibm.com>
Date:   2015-10-27T21:19:17Z

    Add addJar to PySpark API but it isn't very useful at present

commit 3b8b1b086d486c86c0028c19f1875f9e14ea7a53
Author: Holden Karau <ho...@us.ibm.com>
Date:   2015-10-27T22:47:34Z

    Add a helper to update the running class loader when adding classes from python + add a util function to simplify. Follow up should make Kafakutils & related use the util function

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9313#issuecomment-153505891
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-10658][PYSPARK] Provide add jars to py ...

Posted by holdenk <gi...@git.apache.org>.

Github user holdenk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/9313#discussion_r43350330
  
    --- Diff: python/pyspark/context.py ---
    @@ -806,6 +806,24 @@ def addPyFile(self, path):
                 import importlib
                 importlib.invalidate_caches()
     
    +    def addJar(self, path):
    +        """
    +        Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.
    +        The `path` passed can be either a local file, a file in HDFS (or other Hadoop-supported
    +        filesystems), an HTTP, HTTPS or FTP URI, or local:/path for a file on every worker node.
    +        """
    +        self._jsc.sc().addJar(path)
    +        self._jvm.PythonUtils.updatePrimaryClassLoader(self._jsc)
    +
    +    def _loadClass(self, className):
    +        """
    +        .. note:: Experimental
    +
    +        Loads a JVM class using the MutableClass loader used by spark.
    +        This function exists because Py4J uses a different class loader.
    +        """
    +        self._jvm.java.lang.Thread.currentThread().getContextClassLoader().loadClass(className)
    --- End diff --
    
    based on http://www.javaworld.com/article/2077344/core-java/find-a-way-out-of-the-classloader-maze.html every thread has a classloader associated with it unless it was created by native code. This is also the technique @tdas used for getting the class loader in the python kafka utils. Although this did come up (in https://issues.apache.org/jira/browse/SPARK-1403 ). I'll fix it here and make a follow up JIRA to update the kafka utils pyspark to use the common methodlogy.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9313#issuecomment-153474005
  
    **[Test build #44952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44952/consoleFull)** for PR 9313 at commit [`2467ce5`](https://github.com/apache/spark/commit/2467ce59b7a2cf7af71ef896752e88c148116c8e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/9313#issuecomment-153188707
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE] Prov...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9313#issuecomment-154859316
  
    **[Test build #45307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45307/consoleFull)** for PR 9313 at commit [`d9d1375`](https://github.com/apache/spark/commit/d9d13758f4e0b4adaeec07bd2fc9b76a9cbbff0c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-10658][PYSPARK] Provide add jars to py ...

Posted by holdenk <gi...@git.apache.org>.

Github user holdenk commented on the pull request:

    https://github.com/apache/spark/pull/9313#issuecomment-152082830
  
    @sryza it does not, we could make it a flag so the default behaviour is more consistent, although this behaviour is in line with addPyFile which does add the file to the current running context.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-10658][SPARK-11421][PYSPARK][CORE][WIP]...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/9313#issuecomment-152473541
  
    **[Test build #44670 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44670/consoleFull)** for PR 9313 at commit [`371fc7d`](https://github.com/apache/spark/commit/371fc7dc54d68f1cb7e8aa7d3d8d4bcad8838dda).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:\n  * `   * warning. Intended for use by addJar(), although constructing an instance of the class will`\n  * `   * sc._jvm.java.lang.Thread.currentThread().getContextClassLoader().loadClass(\"class name\")`\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org