You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Mingyu Kim <mk...@palantir.com> on 2014/12/19 19:57:41 UTC

Spark master OOMs with exception stack trace stored in JobProgressListener (SPARK-4906)

Hi,

I just filed a bug SPARK-4906<https://issues.apache.org/jira/browse/SPARK-4906>, regarding Spark master OOMs. If I understand correctly, the UI states for all running applications are kept in memory retained by JobProgressListener, and when there are a lot of exception stack traces, this UI states can take up a significant amount of heap. This seems very bad especially for long-running applications.

Can you correct me if I’m misunderstanding anything? If my understanding is correct, is there any work being done to make sure the UI states don’t grow indefinitely over time? Would it make sense to spill some states to disk or work with what spark.eventLog is doing so Spark master doesn’t need to keep things in memory?

Thanks,
Mingyu