[GitHub] [spark] jealous commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks when executors are lost

[GitBox <gi...@apache.org>]

jealous commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks when executors are lost
URL: https://github.com/apache/spark/pull/24462#issuecomment-489482553
 
 
   Hi @Ngone51 , check our project page here:
   https://github.com/MemVerge/splash/
   And the design document here:
   https://github.com/MemVerge/splash/blob/master/doc/Design.md
   The idea is to extract all file/network IO from the shuffle manager.  We supply a shared file system implementation for the storage interface.  But you could always implement your own plugin.
   And we have the same issue, we don't want to re-submit the tasks when the executors are lost.  Because the shuffle files are persisted somewhere else.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org