You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/04/29 04:50:03 UTC
[jira] [Created] (MAPREDUCE-2461) Hudson jobs failing because
mapred staging directory is full
Hudson jobs failing because mapred staging directory is full
------------------------------------------------------------
Key: MAPREDUCE-2461
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2461
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Fix For: 0.23.0
All of the tests that submit MR jobs are failing on the h7 build machine. This is because the staging directory is entirely full:
hudson@h7:/tmp/mr/mr$ ls -l /tmp/hadoop-hudson/mapred/staging/ | wc -l
31999
This makes me think that there's some bug where we're leaking things in the staging directory. I will manually clean this for now, but we should investigate.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2461) Hudson jobs failing because
mapred staging directory is full
Posted by "Thomas Weise (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126732#comment-13126732 ]
Thomas Weise commented on MAPREDUCE-2461:
-----------------------------------------
We see the same issue with 0.20.204.
> Hudson jobs failing because mapred staging directory is full
> ------------------------------------------------------------
>
> Key: MAPREDUCE-2461
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2461
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Fix For: 0.23.0
>
>
> All of the tests that submit MR jobs are failing on the h7 build machine. This is because the staging directory is entirely full:
> hudson@h7:/tmp/mr/mr$ ls -l /tmp/hadoop-hudson/mapred/staging/ | wc -l
> 31999
> This makes me think that there's some bug where we're leaking things in the staging directory. I will manually clean this for now, but we should investigate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2461) Hudson jobs failing because
mapred staging directory is full
Posted by "Dave Latham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dave Latham updated MAPREDUCE-2461:
-----------------------------------
Affects Version/s: 1.0.2
> Hudson jobs failing because mapred staging directory is full
> ------------------------------------------------------------
>
> Key: MAPREDUCE-2461
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2461
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0, 1.0.2
> Reporter: Todd Lipcon
> Fix For: 0.24.0
>
>
> All of the tests that submit MR jobs are failing on the h7 build machine. This is because the staging directory is entirely full:
> hudson@h7:/tmp/mr/mr$ ls -l /tmp/hadoop-hudson/mapred/staging/ | wc -l
> 31999
> This makes me think that there's some bug where we're leaking things in the staging directory. I will manually clean this for now, but we should investigate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2461) Hudson jobs failing because
mapred staging directory is full
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271984#comment-13271984 ]
Luke Lu commented on MAPREDUCE-2461:
------------------------------------
We ran into this issue as well. The problem is from the staging area dirs from jobs in local mode (via LocalJobRunner), which creates a staging area dir as <staging_root>/<user><random>/.staging instead of just <staging_root>/<user>/.staging in cluster mode. The issue is introduced with the security releases (since 0.20.20x) when the getStageAreaDir API is introduced.
The random number for the local mode is presumably used to avoid job collisions, since there is no jobtracker to issue unique job ids.
Mayb we can introduce a feature (mapreduce.job.staging.keep=<number of latest jobs to keep>) to prune these directories once in a while.
> Hudson jobs failing because mapred staging directory is full
> ------------------------------------------------------------
>
> Key: MAPREDUCE-2461
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2461
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Fix For: 0.24.0
>
>
> All of the tests that submit MR jobs are failing on the h7 build machine. This is because the staging directory is entirely full:
> hudson@h7:/tmp/mr/mr$ ls -l /tmp/hadoop-hudson/mapred/staging/ | wc -l
> 31999
> This makes me think that there's some bug where we're leaking things in the staging directory. I will manually clean this for now, but we should investigate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira