Spark on Mesos cause mesos-master OOM

[Chengwei Yang <ch...@gmail.com>]

Hi List,

We're recently trying to running spark on Mesos, however, we encountered a fatal
error that mesos-master process will continuousely consume memory and finally
killed by OOM Killer, this situation only happening if has spark job
(fine-grained mode) running.

We finally root caused the issue and found that is because spark executor attach
rdd computed result in TaskStatus, like below:

---------------------->8------------------------------->8-------------
spark.git/core/src/main/scala/org/apache/spark/executor/Executor.scala

        val serializedDirectResult = ser.serialize(directResult)
        logInfo("Serialized size of result for " + taskId + " is " + serializedDirectResult.limit)

        val serializedResult = {
          if (serializedDirectResult.limit >= execBackend.akkaFrameSize() -
              AkkaUtils.reservedSizeBytes) {
            logInfo("Storing result for " + taskId + " in local BlockManager")
            val blockId = TaskResultBlockId(taskId)
            env.blockManager.putBytes(
              blockId, serializedDirectResult, StorageLevel.MEMORY_AND_DISK_SER)
            ser.serialize(new IndirectTaskResult[Any](blockId))
          } else {
            logInfo("Sending result for " + taskId + " directly to driver")
            serializedDirectResult
          }
        }

        execBackend.statusUpdate(taskId, TaskState.FINISHED, serializedResult)
        logInfo("Finished task ID " + taskId)
------------------->8-------------------------8>---------------------

And the spark executor log says how large the serializedResult is like below:

14/08/22 13:29:18 INFO Executor: Serialized size of result for 248 is 17573033

Since in fine-grained mode, every singe spark stage finished in say 10 seconds
and may have tens of tasks, so it's generally fail mesos-master OOM in tens of
minutes.

I'm not familiar with spark, and I'm wondering if we should not store
serializedResult into TaskStatus?

--
Thanks,
Chengwei