You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Ashwin Shankar (JIRA)" <ji...@apache.org> on 2015/08/03 21:18:05 UTC
[jira] [Updated] (YARN-4011) Jobs fail since nm-local-dir not
cleaned up when rogue job fills up disk
[ https://issues.apache.org/jira/browse/YARN-4011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashwin Shankar updated YARN-4011:
---------------------------------
Description:
We observed jobs failed since tasks couldn't launch on nodes due to "java.io.IOException No space left on device".
On digging in further, we found a rogue job which filled up disk.
Specifically it was wrote a lot of map spills(like attempt_1432082376223_461647_m_000421_0_spill_10000.out) to nm-local-dir causing disk to fill up, and it failed/got killed, but didn't clean up its files in nm-local-dir.
So the disk remained full, causing subsequent jobs to fail.
This jira is created to address why files under nm-local-dir doesn't get cleaned up when job fails after filling up disk.
was:
We observed jobs failed since tasks couldn't launch on nodes due to "java.io.IOException No space left on device".
On digging in further, we found a rogue job which filled up disk.
Specifically it was wrote a lot of map spills(like attempt_1432082376223_461647_m_000421_0_spill_10000.out) to nm-local-dir causing disk to fill up, and it failed/got killed, but didn't clean up its files in nm-local-dir.
So the disk remained full, causing subsequent jobs to fail.
> Jobs fail since nm-local-dir not cleaned up when rogue job fills up disk
> ------------------------------------------------------------------------
>
> Key: YARN-4011
> URL: https://issues.apache.org/jira/browse/YARN-4011
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Affects Versions: 2.4.0
> Reporter: Ashwin Shankar
>
> We observed jobs failed since tasks couldn't launch on nodes due to "java.io.IOException No space left on device".
> On digging in further, we found a rogue job which filled up disk.
> Specifically it was wrote a lot of map spills(like attempt_1432082376223_461647_m_000421_0_spill_10000.out) to nm-local-dir causing disk to fill up, and it failed/got killed, but didn't clean up its files in nm-local-dir.
> So the disk remained full, causing subsequent jobs to fail.
> This jira is created to address why files under nm-local-dir doesn't get cleaned up when job fails after filling up disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)