failed mesos task loses executor

[Adrian Bridgett <ad...@opensignal.com>]

Just testing spark v1.5.0 (on mesos v0.23) and we saw something 
unexpected (according to the event timeline) - when a spark task failed 
(intermittent S3 connection failure), the whole executor was removed and 
was never recovered so the job proceeded slower than normal.

Looking at the code I saw something that seemed a little odd in 
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala:

   override def statusUpdate(d: SchedulerDriver, status: TaskStatus) {
...
         if (TaskState.isFailed(TaskState.fromMesos(status.getState))
           && taskIdToSlaveId.contains(tid)) {
           // We lost the executor on this slave, so remember that it's gone
           removeExecutor(taskIdToSlaveId(tid), "Lost executor")
         }
         if (TaskState.isFinished(state)) {
           taskIdToSlaveId.remove(tid)
         }
       }

I don't know either codebase at all, however it seems odd to kill the 
executor for a failed task rather than just a lost task.  I did a quick 
test (with v1.5.1) where I replaced this line with:

    if ((TaskState.fromMesos(status.getState) == TaskState.LOST)

and all seemed well - I faked the problem (using iptables to briefly 
block access to the S3 endpoint), the task failed but was retried (on 
the same executor), succeeded and continued on its merry way.

Adrian
-- 
*Adrian Bridgett* |  Sysadmin Engineer, OpenSignal 
<http://www.opensignal.com>
_____________________________________________________
Office: First Floor, Scriptor Court, 155-157 Farringdon Road, 
Clerkenwell, London, EC1R 3AD
Phone #: +44 777-377-8251
Skype: abridgett  |@adrianbridgett <http://twitter.com/adrianbridgett>| 
LinkedIn link <https://uk.linkedin.com/in/abridgett>
_____________________________________________________