[jira] [Updated] (SPARK-44639) Fix to use Java tmp dir for local RocksDB state storage on Yarn

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-44639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun updated SPARK-44639:
----------------------------------
    Summary: Fix to use Java tmp dir for local RocksDB state storage on Yarn  (was: Add option to use Java tmp dir for RocksDB state store)

> Fix to use Java tmp dir for local RocksDB state storage on Yarn
> ---------------------------------------------------------------
>
>                 Key: SPARK-44639
>                 URL: https://issues.apache.org/jira/browse/SPARK-44639
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL, Structured Streaming
>    Affects Versions: 3.4.1
>            Reporter: Adam Binford
>            Assignee: Adam Binford
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.1.0
>
>
> Currently local RocksDB state is stored in a local directory given by Utils.getLocalDir. On yarn this is a directory created inside the root application folder such as
> {{/tmp/nm-local-dir/usercache/<user>/appcache/<app_id>/}}
> The problem with this is that if an executor crashes for some reason (like OOM) and the shutdown hooks don't get run, this directory will stay around forever until the application finishes, which can cause jobs to slowly accumulate more and more temporary space until finally the node manager goes unhealthy.
> Because this data will only ever be accessed by the executor that created this directory, it would make sense to store the data inside the container folder, which will always get cleaned up by the node manager when that yarn container gets cleaned up. Yarn sets the `java.io.tmpdir` to a path inside this directory, such as
> {{/tmp/nm-local-dir/usercache/<user>/appcache/<app_id>/<container_id>/tmp/}}
> I'm not sure the behavior for other resource managers, so this could be an opt-in config that can be specified.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org