[GitHub] clems4ever opened a new pull request #23758: [SPARK-17454][MESOS] Use Mesos disk resources for executors.

[GitBox <gi...@apache.org>]

clems4ever opened a new pull request #23758: [SPARK-17454][MESOS] Use Mesos disk resources for executors.
URL: https://github.com/apache/spark/pull/23758
 
 
   ## What changes were proposed in this pull request?
   
   Before this change, there was no way to allocate a given amount of
   disk when using Mesos scheduler. It's good enough when using default isolation
   options but not when enabling the XFS isolator with hard limit in order to
   properly isolate all containers. In that case, the executor is killed by Mesos
   during the download of the Spark executor archive.
   
   Therefore, this change introduces a configuration flag, specific to Mesos, to
   declare the amount of disk required by the executors and therefore prevent
   Mesos from killing the container because the XFS hard limit has been exceeded.
   
   ## How was this patch tested?
   
   I added 3 unit tests and tested my built version of Spark against a real Mesos cluster.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org