[jira] [Issue Comment Deleted] (SPARK-671) Spark runs out of memory on fork/exec (affects both pipes and python)

["hotdog (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

hotdog updated SPARK-671:
-------------------------
    Comment: was deleted

(was: is it still a problem? )

> Spark runs out of memory on fork/exec (affects both pipes and python)
> ---------------------------------------------------------------------
>
>                 Key: SPARK-671
>                 URL: https://issues.apache.org/jira/browse/SPARK-671
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark, Spark Core
>    Affects Versions: 0.5.0, 0.5.1, 0.5.2, 0.6.0, 0.6.1
>            Reporter: Patrick Cogan
>            Assignee: Jey Kottalam
>             Fix For: 0.7.3, 0.8.0
>
>
> Because the JVM uses fork/exec to launch child processes, any child process initially has the memory footprint of its parent. In the case of a large Spark JVM that spawns many child processes (for Pipe or Python support), this quickly leads to kernel memory exhaustion.
> This problem is discussed here:
> https://gist.github.com/1970815
> It results in errors like this:
> {code}
> 13/01/31 20:18:48 INFO cluster.TaskSetManager: Loss was due to java.io.IOException: Cannot run program "cat": java.io.IOException: error=12, Cannot allocate memory
>        at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
>        at spark.rdd.PipedRDD.compute(PipedRDD.scala:38)
>        at spark.RDD.computeOrReadCheckpoint(RDD.scala:203)
>        at spark.RDD.iterator(RDD.scala:192)
>        at spark.scheduler.ResultTask.run(ResultTask.scala:76)
> {code}
> I was able to workaround by allowing for memory over-commitment by the kernel on all slaves,
> {code}
> echo 1 > /proc/sys/vm/overcommit_memory
> {code}
> but we should try to include a more robust solution, such as the one here:
> https://github.com/axiak/java_posix_spawn



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org