[mesos] dispatcher and mesos constraints applied to driver - SPARK-19606

[Paul Mackles <pm...@adobe.com.INVALID>]

Hi - I was hoping to revive the discussion regarding the application of mesos constraints to driver tasks. I added some comments to: <https://issues.apache.org/jira/browse/>

https://issues.apache.org/jira/browse/SPARK-19606

Basically, I am proposing the introduction of an additional optional configuration property named "spark.mesos.constraints.driver" which would function the same as "spark.mesos.constraints" except that it would be applied to driver tasks by the mesos dispatcher.

Aside from being backwards compatible, it would allow driver tasks to be constrained to either the same or different resources as the executors.

This would provide better support for multi-tenant cluster as well as provide a means for limiting the number of drivers that are allowed to run concurrently on a cluster since too many drivers can lead to starvation scenarios.

If folks agree that this is a good change, I am about 75% done with a PR to implement the above. There is an alternative proposal (and PR) in the ticket as well if that is the preferred approach.

Thanks,
Paul