You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/10 13:56:05 UTC

[GitHub] [spark] tgravescs commented on pull request #29378: [SPARK-30069][CORE][YARN] Clean up non-shuffle disk block manager files following executor exists on YARN

tgravescs commented on pull request #29378:
URL: https://github.com/apache/spark/pull/29378#issuecomment-671369062


   so I haven't looked at this fully but from skimming it, it seems like we are introducing yarn specific concepts into the disk block manager. Maybe its just the wording of calling it container dir for instance, but I would like to see as much yarn specific stuff not in there.  At least put it in Utils like we have getYarnLocalDirs. Also perhaps just name it something generic like temp file directories.  Can this be reused by standalone mode and others so we keep that logic similar between the cluster managers?  The standalone fix looks like it recurses all the files looking for those names, if it could just remove a single directory would seem more efficient but I would have to look more in depth.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org