[GitHub] [spark] tgravescs edited a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone

[GitBox <gi...@apache.org>]

tgravescs edited a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592042018
 
 
   >  Ideally, it should be an edge node out of cluster for the driver and assumed not to be on the same of host of worker.
   
   This may be ideal but not true in many cases, especially in just test/dev setups, but its no different from yarn case with multiple drivers so can see this being ok.
   
   For me this comes down to making things convenient for the user and doesn't seem like much overhead on the Spark side, especially since its already implemented/tested.  If you aren't using custom resources, the code has no affect (or shouldn't anyway - if it does we should fix). Its configurable off if people don't want it or see some overhead from the file locking pieces. The code for this in my opinion is not that much and pretty isolated. I don't see much downside here.
   
   I guess the spark.resources.dir could be left around in worker crash type cases. Though again could be turned off. 
   
   The only way I see to fix the local-cluster mode is leave this code or somehow allow users to specify different resource files or script per worker, which seems like a hard/weird api. 
   
   I would like more justification here? why is this a high maintenance cost? the code is done and tested and I don't envision it needed more feature work. local-cluster mode unit tests should be enough to test it pretty thoroughly (perhaps we need more).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org