[jira] [Resolved] (SPARK-1700) PythonRDD leaks socket descriptors during cancellation

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-1700.
------------------------------
       Resolution: Fixed
    Fix Version/s: 1.0.0

The PR was https://github.com/apache/spark/pull/623 and says it was merged in 1.0.

> PythonRDD leaks socket descriptors during cancellation
> ------------------------------------------------------
>
>                 Key: SPARK-1700
>                 URL: https://issues.apache.org/jira/browse/SPARK-1700
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 0.9.0, 1.0.0
>            Reporter: Aaron Davidson
>            Assignee: Aaron Davidson
>             Fix For: 1.0.0
>
>
> Sockets from Spark to Python workers are not cleaned up over the duration of a job, causing the total number of opened file descriptors to grow to around the number of partitions in the job. Usually these go away if the job is successful, but in the case of cancellation (and possibly exceptions, though I haven't investigated), the socket file descriptors remain indefinitely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org