You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Vinod K V (JIRA)" <ji...@apache.org> on 2009/10/23 11:40:59 UTC
[jira] Created: (MAPREDUCE-1141) Localization of a task's
distributed-cache files gets blocked by deletion of old unrelated files
when mapreduce.tasktracker.cache.local.size is hit.
Localization of a task's distributed-cache files gets blocked by deletion of old unrelated files when mapreduce.tasktracker.cache.local.size is hit.
----------------------------------------------------------------------------------------------------------------------------------------------------
Key: MAPREDUCE-1141
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1141
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Affects Versions: 0.20.2, 0.21.0, 0.22.0
Reporter: Vinod K V
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1141) Localization of a task's
distributed-cache files gets blocked by deletion of old unrelated files
when mapreduce.tasktracker.cache.local.size is hit.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V resolved MAPREDUCE-1141.
----------------------------------
Resolution: Duplicate
MAPREDUCE-1302 already started implementation for this issue. Closing this as duplicate.
> Localization of a task's distributed-cache files gets blocked by deletion of old unrelated files when mapreduce.tasktracker.cache.local.size is hit.
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1141
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Vinod K V
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1141) Localization of a task's
distributed-cache files gets blocked by deletion of old unrelated files
when mapreduce.tasktracker.cache.local.size is hit.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769151#action_12769151 ]
Vinod K V commented on MAPREDUCE-1141:
--------------------------------------
Here's roughly the offending code:
{code}
getLocalizeCache(URI uri) {
localizeCache();
if(currentDiskUsage > conf.get(mapreduce.tasktracker.cache.local.size)) {
deleteOldFilesInCache();
}
}
{code}
So, when mapreduce.tasktracker.cache.local.size is hit, localization of a task will not complete _till_ the deleltion of (possibly a large number of) old DistributedCache files is completed (which can possibly take long time).
Localization of a task should not ideally be blocked on cleanup of old files. Clean-up can best be done in a separate thread.
> Localization of a task's distributed-cache files gets blocked by deletion of old unrelated files when mapreduce.tasktracker.cache.local.size is hit.
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1141
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1141
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Vinod K V
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.