You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-issues@hadoop.apache.org by "Shilun Fan (Jira)" <ji...@apache.org> on 2023/10/30 23:00:00 UTC

[jira] [Resolved] (MAPREDUCE-7457) Limit number of spill files getting created

     [ https://issues.apache.org/jira/browse/MAPREDUCE-7457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shilun Fan resolved MAPREDUCE-7457.
-----------------------------------
       Fix Version/s: 3.4.0
        Hadoop Flags: Reviewed
    Target Version/s: 3.4.0
          Resolution: Fixed

> Limit number of spill files getting created
> -------------------------------------------
>
>                 Key: MAPREDUCE-7457
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7457
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Mudit Sharma
>            Priority: Critical
>              Labels: pull-request-available
>             Fix For: 3.4.0
>
>
> Hi,
>  
> We have been facing some issues where many of our cluster node disks go full because of some rogue applications creating a lot of spill data
> We wanted to fail the app if more than a threshold amount of spill files are written
> Please let us know if any such capability is supported
>  
> If the capability is not there, we are proposing it to support it via a config, we have added a PR for the same: [https://github.com/apache/hadoop/pull/6155]  please let us know your thoughts on it



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org