You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2014/07/31 06:13:38 UTC

[jira] [Assigned] (SPARK-2764) Simplify process structure of PySpark daemon / worker launching process

     [ https://issues.apache.org/jira/browse/SPARK-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Josh Rosen reassigned SPARK-2764:
---------------------------------

    Assignee: Josh Rosen

> Simplify process structure of PySpark daemon / worker launching process
> -----------------------------------------------------------------------
>
>                 Key: SPARK-2764
>                 URL: https://issues.apache.org/jira/browse/SPARK-2764
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>            Reporter: Josh Rosen
>            Assignee: Josh Rosen
>
> PySpark's daemon-based worker factory has a very complicated process structure that I've always found confusing.  The per-java-worker daemon.py process launches a numCores-sized pool of subprocesses, and those subprocesses launching the actual worker processes that process data.
> I think we can simplify this by having daemon.py launch the workers directly without this extra layer of indirection.  See my comments on the pull request that introduced daemon.py: https://github.com/mesos/spark/pull/563



--
This message was sent by Atlassian JIRA
(v6.2#6252)