You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by vundela <gi...@git.apache.org> on 2017/04/21 18:45:39 UTC

[GitHub] spark pull request #17722: [SPARK-12717][PYSPARK][BRANCH-1.6] Resolving race...

GitHub user vundela opened a pull request:

    https://github.com/apache/spark/pull/17722

    [SPARK-12717][PYSPARK][BRANCH-1.6] Resolving race condition with pyspark broadcasts when using multiple threads

    
    
    ## What changes were proposed in this pull request?
    
    In pyspark when multiple threads are used, broadcast variables are picked with wrong PythonRDD wrap functions which leads to the following exception(Because of the race condition between the threads on java side with py4j).
    
    16/01/08 17:10:20 ERROR Executor: Exception in task 0.0 in stage 9.0 (TID 9)
    org.apache.spark.api.python.PythonException: Traceback (most recent call last):
    File "/Network/Servers/mother.adverplex.com/Volumes/homeland/Users/walker/.spark/spark-1.6.0-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/worker.py", line 98, in main
    command = pickleSer._read_with_length(infile)
    File "/Network/Servers/mother.adverplex.com/Volumes/homeland/Users/walker/.spark/spark-1.6.0-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/serializers.py", line 164, in _read_with_length
    return self.loads(obj)
    File "/Network/Servers/mother.adverplex.com/Volumes/homeland/Users/walker/.spark/spark-1.6.0-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/serializers.py", line 422, in loads
    return pickle.loads(obj)
    File "/Network/Servers/mother.adverplex.com/Volumes/homeland/Users/walker/.spark/spark-1.6.0-bin-hadoop2.6/python/lib/pyspark.zip/pyspark/broadcast.py", line 39, in _from_id
    raise Exception("Broadcast variable '%s' not loaded!" % bid)
    Exception: (Exception("Broadcast variable '6' not loaded!",), <function _from_id at 0xce7a28>, (6L,))
    
    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)
    at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:207)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    
    This change will fix the race condition in branch-1.6 by making sure that broadcast variables are picked with same pythonRDD function.
    
    ## How was this patch tested?
    Reproduced the issue mentioned in SPARK-12717, following the instructions specified in jira
    Make sure that issue is fixed with the changes
    
    Please review http://spark.apache.org/contributing.html before opening a pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/vundela/spark SPARK-12717_spark1.6

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/17722.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #17722
    
----
commit af8e6d762a5ce481ab5cb0a385660e2766bee195
Author: Srinivasa Reddy Vundela <vs...@cloudera.com>
Date:   2017-04-21T18:31:51Z

    [SPARK-12717][PYSPARK] Resolving race condition with pyspark broadcasts when using multiple threads

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17722: [SPARK-12717][PYSPARK][BRANCH-1.6] Resolving race condit...

Posted by maver1ck <gi...@git.apache.org>.

Github user maver1ck commented on the issue:

    https://github.com/apache/spark/pull/17722
  
    Hi,
    What about this issue ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #17722: [SPARK-12717][PYSPARK][BRANCH-1.6] Resolving race...

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/17722


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17722: [SPARK-12717][PYSPARK][BRANCH-1.6] Resolving race condit...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/17722
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #17722: [SPARK-12717][PYSPARK][BRANCH-1.6] Resolving race condit...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/17722
  
    One thing I am sure is, @BryanCutler proposed, IMHO, a better approach and merged into master, 2.2 and 2.1. I guess this PR should be closed at least.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org