You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Davies Liu (JIRA)" <ji...@apache.org> on 2014/09/11 02:57:34 UTC

[jira] [Created] (SPARK-3478) Profile Python tasks stage by stage in worker

Davies Liu created SPARK-3478:
---------------------------------

             Summary: Profile Python tasks stage by stage in worker
                 Key: SPARK-3478
                 URL: https://issues.apache.org/jira/browse/SPARK-3478
             Project: Spark
          Issue Type: New Feature
          Components: PySpark
            Reporter: Davies Liu


The Python code in driver is easy to profile by users, but the code run in worker is distributed in clusters, is not easy to profile by users.

So we need a way to do the profiling in worker and aggregate all the result together for users.

This also can be used to analys the bottleneck in PySpark.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org