pyspark exiting cleanly, but executor remains on slave as an orphaned task

[June Taylor <ju...@umn.edu>]

We are running pyspark against our cluster in coarse-grained mode by
specifying the --master mesos://host:5050 flag, which properly creates one
task on each node.

However, if the driver is shut down, it appears that these executors become
orphaned_tasks, still consuming resources on the slave, but no longer being
represented in the master's understanding of available resources.

Examining the stdout/stderr shows it exited:

Registered executor on node4
Starting task 0
sh -c 'cd spark-1*;  ./bin/spark-class
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://
CoarseGrainedScheduler@128.101.163.200:41563 --executor-id
aa1337b6-43b0-4236-b445-c8ccbfb60506-S2/0 --hostname node4 --cores 31
--app-id aa1337b6-43b0-4236-b445-c8ccbfb60506-0097'
Forked command at 117620
Command exited with status 1 (pid: 117620)

But, these executors are remaining on all the slaves.

What can we do to clear them out? Stopping mesos-slave and removing the
full work-dir is successful, but also destroys our other tasks.

Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota