[jira] [Created] (SPARK-44639) Add option to use Java tmp dir for RocksDB state store

["Adam Binford (Jira)" <ji...@apache.org>]

Adam Binford created SPARK-44639:
------------------------------------

             Summary: Add option to use Java tmp dir for RocksDB state store
                 Key: SPARK-44639
                 URL: https://issues.apache.org/jira/browse/SPARK-44639
             Project: Spark
          Issue Type: Improvement
          Components: SQL, Structured Streaming
    Affects Versions: 3.4.1
            Reporter: Adam Binford


Currently local RocksDB state is stored in a local directory given by Utils.getLocalDir. On yarn this is a directory created inside the root application folder such as

{{/tmp/nm-local-dir/usercache/<user>/appcache/<app_id>/}}

The problem with this is that if an executor crashes for some reason (like OOM) and the shutdown hooks don't get run, this directory will stay around forever until the application finishes, which can cause jobs to slowly accumulate more and more temporary space until finally the node manager goes unhealthy.

Because this data will only ever be accessed by the executor that created this directory, it would make sense to store the data inside the container folder, which will always get cleaned up by the node manager when that yarn container gets cleaned up. Yarn sets the `java.io.tmpdir` to a path inside this directory, such as

{{/tmp/nm-local-dir/usercache/<user>/appcache/<app_id>/<container_id>/tmp/}}

I'm not sure the behavior for other resource managers, so this could be an opt-in config that can be specified.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org