You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Tal Sliwowicz (JIRA)" <ji...@apache.org> on 2014/10/20 10:42:34 UTC

[jira] [Created] (SPARK-4006) Spark Driver crashes whenever an Executor is registered twice

Tal Sliwowicz created SPARK-4006:
------------------------------------

             Summary: Spark Driver crashes whenever an Executor is registered twice
                 Key: SPARK-4006
                 URL: https://issues.apache.org/jira/browse/SPARK-4006
             Project: Spark
          Issue Type: Bug
          Components: Block Manager, Spark Core
    Affects Versions: 1.1.0, 1.0.2, 0.9.2
         Environment: Mesos, Coarse Grained
            Reporter: Tal Sliwowicz
            Priority: Critical


We have long running spark drivers and even though we have state of the art hardware, from time to time executors disconnect. In many cases, the RemoveExecutor is not received, and when the new executor registers, the driver crashes. In mesos coarse grained, executor ids are fixed. 

The issue is with the System.exit(1) in BlockManagerMasterActor


private def register(id: BlockManagerId, maxMemSize: Long, slaveActor: ActorRef) {
    if (!blockManagerInfo.contains(id)) {
      blockManagerIdByExecutor.get(id.executorId) match {
        case Some(manager) =>
          // A block manager of the same executor already exists.
          // This should never happen. Let's just quit.
          logError("Got two different block manager registrations on " + id.executorId)
          System.exit(1)
        case None =>
          blockManagerIdByExecutor(id.executorId) = id
      }

      logInfo("Registering block manager %s with %s RAM".format(
        id.hostPort, Utils.bytesToString(maxMemSize)))

      blockManagerInfo(id) =
        new BlockManagerInfo(id, System.currentTimeMillis(), maxMemSize, slaveActor)
    }
    listenerBus.post(SparkListenerBlockManagerAdded(id, maxMemSize))
  }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org