You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2025/11/20 17:59:03 UTC
Re: [PR] [SPARK-44639][SS][YARN] Use Java tmp dir for local RocksDB state storage on Yarn [spark]
dongjoon-hyun commented on PR #42301:
URL: https://github.com/apache/spark/pull/42301#issuecomment-3559346468
Thank you for updating your PR, @Kimahriman .
To @pan3793 , IIRC, you are using [YARN in a way](https://github.com/apache/spark/pull/52923#discussion_r2501639415). I'm wondering how you handle this RocksDB issue (in this PR description)?
> The problem with this is that if an executor crashes for some reason (like OOM) and the shutdown hooks don't get run, this directory will stay around forever until the application finishes, which can cause jobs to slowly accumulate more and more temporary space until finally the node manager goes unhealthy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org