You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Nathan Kronenfeld (JIRA)" <ji...@apache.org> on 2014/12/05 22:50:12 UTC

[jira] [Created] (SPARK-4772) Accumulators leak memory, both temporarily and permanently

Nathan Kronenfeld created SPARK-4772:
----------------------------------------

             Summary: Accumulators leak memory, both temporarily and permanently
                 Key: SPARK-4772
                 URL: https://issues.apache.org/jira/browse/SPARK-4772
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
            Reporter: Nathan Kronenfeld
             Fix For: 1.3.0


Accumulators.localAccums is cleared at the beginning of a task, and not at the end.

This means that any locally accumulated values hang around until another task is run on that thread.

If for some reason, the thread dies, said values hang around indefinitely.

This is really only a big issue with very large accumulators.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org