You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Davis Shepherd (JIRA)" <ji...@apache.org> on 2014/12/05 08:44:12 UTC
[jira] [Created] (SPARK-4759) Deadlock in pathological spark job.
Davis Shepherd created SPARK-4759:
-------------------------------------
Summary: Deadlock in pathological spark job.
Key: SPARK-4759
URL: https://issues.apache.org/jira/browse/SPARK-4759
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.1.1
Environment: Java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
Mac OSX 10.10.1
Using local spark context
Reporter: Davis Shepherd
The attached test class runs two identical jobs that perform some iterative computation on and RDD[Int, Int]. This computation involves
taking new data merging it with the previous result
caching and checkpointing the new result
rinse and repeat
The first time the job is run, it runs successfully, and the spark context is shut down. The second time the job is run with a new spark context in the same process, the job hangs indefinitely, only having scheduled a subset of the necessary tasks for the final stage.
Ive been able to produce a test case that reproduces the issue, and I've added some comments where some knockout experimentation has left some breadcrumbs as to where the issue might be.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org