[jira] [Resolved] (IMPALA-9697) Support priority based scratch directory selection

["Abhishek Rawat (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-9697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Abhishek Rawat resolved IMPALA-9697.
------------------------------------
    Resolution: Fixed

> Support priority based scratch directory selection 
> ---------------------------------------------------
>
>                 Key: IMPALA-9697
>                 URL: https://issues.apache.org/jira/browse/IMPALA-9697
>             Project: IMPALA
>          Issue Type: Task
>            Reporter: Abhishek Rawat
>            Assignee: Abhishek Rawat
>            Priority: Major
>             Fix For: Impala 4.0
>
>
> The `‑‑scratch_dirs` startup flag uses the given scratch directories in a round robin manner. This may not always be ideal since these directories could come from different class of storage system volumes having different performance characteristics (SSD vs HDD, local storage vs network attached storage, etc.). Giving user an option to configure the priority of their scratch directories could help them optimize their workload based on their storage system configuration.
> One possible way could be that the user pass the priority as part of the `–scratch_dirs` startup flag using <directory>:<spill_priority>. The directories will be selected for spilling based on their priorities and if multiple directories have the same priority then they will be selected in a round robin fashion. In the below example, dir1 will be used as a spill victim until its full and then dir2, dir3, and dir4 will be used in a round robin fashion.
> {code:java}
> ‑‑scratch_dirs="dir1:200GB:0, dir2:1024GB:1, dir3:1024GB:1, dir4:1024GB:1"{code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)