[jira] [Created] (SPARK-3334) Spark causes mesos-master memory leak

["Iven Hsu (JIRA)" <ji...@apache.org>]

Iven Hsu created SPARK-3334:
-------------------------------

             Summary: Spark causes mesos-master memory leak
                 Key: SPARK-3334
                 URL: https://issues.apache.org/jira/browse/SPARK-3334
             Project: Spark
          Issue Type: Bug
          Components: Mesos
    Affects Versions: 1.0.2
         Environment: Mesos 0.16.0/0.19.0
CentOS 6.4
            Reporter: Iven Hsu


The {{akkaFrameSize}} is set to {{Long.MaxValue}} in MesosBackend to workaround SPARK-1112, this causes all serialized task result is sent using Mesos TaskStatus.

mesos-master stores TaskStatus in memory, and when running Spark, it's memory grows very fast, and will be OOM killed.

See MESOS-1746 for more.

I've tryed to set {{akkaFrameSize}} to 0, mesos-master won't be killed, however, the driver will block after success unless I use {{sc.stop()}} to quit it manually. Not sure if it's related to SPARK-1112.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org