You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Milind Bhandarkar (Commented) (JIRA)" <ji...@apache.org> on 2011/11/03 00:13:34 UTC
[jira] [Commented] (MAPREDUCE-1100) User's task-logs filling up
local disks on the TaskTrackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142664#comment-13142664 ]
Milind Bhandarkar commented on MAPREDUCE-1100:
----------------------------------------------
I took a look at this and other jira's to make it work with Hadoop 0.22. I think that based on the multiple dependencies, this will require a lot of changes to be pulled in from the 0.20.2xx branch, and is not worth the risk. I know of at least a handful of production deployments that have circumvented this with a simple cron job looking at log dirs and doing cleanups outside of the framework, and do not this this to be a blocker.
Konstantin, it's your call now.
> User's task-logs filling up local disks on the TaskTrackers
> -----------------------------------------------------------
>
> Key: MAPREDUCE-1100
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1100
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.21.1, 0.22.0
>
> Attachments: MAPREDUCE-1100-20091102.txt, MAPREDUCE-1100-20091106.txt, MAPREDUCE-1100-20091216.2.txt, patch-1100-fix-ydist.2.txt, reducetask-log-level.patch
>
>
> Some user's jobs are filling up TT disks by outrageous logging. mapreduce.task.userlog.limit.kb is not enabled on the cluster. Disks are getting filled up before task-log cleanup via mapred.task.userlog.retain.hours can kick in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira