You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Davis Shepherd (JIRA)" <ji...@apache.org> on 2014/04/04 02:40:14 UTC

[jira] [Created] (SPARK-1411) When using spark.ui.retainedStages=n only the first n stages are kept, not the most recent.

Davis Shepherd created SPARK-1411:
-------------------------------------

             Summary: When using spark.ui.retainedStages=n only the first n stages are kept, not the most recent.
                 Key: SPARK-1411
                 URL: https://issues.apache.org/jira/browse/SPARK-1411
             Project: Spark
          Issue Type: Bug
          Components: Web UI
    Affects Versions: 0.9.0
         Environment: Ubuntu 12.04 LTS Precise
4 Nodes, 
16 cores each, 
196GB RAM each.
            Reporter: Davis Shepherd
            Priority: Minor


For any long running job with many stages, the web ui only shows the first n stages of the job (where spark.ui.retainedStages=n). The most recent stages are immediately dropped and are only visible for a brief time.  This renders the UI pretty useless after a pretty short amount of time for a long running non-streaming job. I am unsure as to whether similar results appear for streaming jobs.



--
This message was sent by Atlassian JIRA
(v6.2#6252)