You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Runping Qi (JIRA)" <ji...@apache.org> on 2008/08/27 03:14:44 UTC
[jira] Created: (HADOOP-4033) Task tracker should not ask for tasks
if its temp disk space is almost full
Task tracker should not ask for tasks if its temp disk space is almost full
---------------------------------------------------------------------------
Key: HADOOP-4033
URL: https://issues.apache.org/jira/browse/HADOOP-4033
Project: Hadoop Core
Issue Type: Bug
Affects Versions: 0.17.1
Reporter: Runping Qi
I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
Consequently, it had hard time to finish the tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4033) Task tracker should not ask for
tasks if its temp disk space is almost full
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668789#action_12668789 ]
Sameer Paranjpye commented on HADOOP-4033:
------------------------------------------
It should also be possible to set minspacestart to a reasonable value in mapred-site.xml, no?
> Task tracker should not ask for tasks if its temp disk space is almost full
> ---------------------------------------------------------------------------
>
> Key: HADOOP-4033
> URL: https://issues.apache.org/jira/browse/HADOOP-4033
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.1
> Reporter: Runping Qi
> Assignee: Ravi Gummadi
>
> I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
> Consequently, it had hard time to finish the tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4033) Task tracker should not ask for
tasks if its temp disk space is almost full
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668795#action_12668795 ]
Ravi Gummadi commented on HADOOP-4033:
--------------------------------------
Devaraj suggested porting JIRA-657's patch to branch 18 would be better. It applies fine to branch 18. Looking into the code of that patch to understand what is happening.
> Task tracker should not ask for tasks if its temp disk space is almost full
> ---------------------------------------------------------------------------
>
> Key: HADOOP-4033
> URL: https://issues.apache.org/jira/browse/HADOOP-4033
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.1
> Reporter: Runping Qi
> Assignee: Ravi Gummadi
>
> I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
> Consequently, it had hard time to finish the tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4033) Task tracker should not ask for
tasks if its temp disk space is almost full
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12625939#action_12625939 ]
Runping Qi commented on HADOOP-4033:
------------------------------------
In the case where multiple temp dirs are configured, hadoop should avoid using a dir when it's available space is below certain threshold.
> Task tracker should not ask for tasks if its temp disk space is almost full
> ---------------------------------------------------------------------------
>
> Key: HADOOP-4033
> URL: https://issues.apache.org/jira/browse/HADOOP-4033
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.1
> Reporter: Runping Qi
>
> I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
> Consequently, it had hard time to finish the tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4033) Task tracker should not ask for
tasks if its temp disk space is almost full
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665787#action_12665787 ]
Ravi Gummadi commented on HADOOP-4033:
--------------------------------------
Configuring mapred.local.dir.minspacestart by user should solve this problem.
Currently, the default value of mapred.local.dir.minspacestart is 0 bytes. How about changing this default value to a +ve value(say 1MB) in mapred.default.xml ?
Thoughts ?
> Task tracker should not ask for tasks if its temp disk space is almost full
> ---------------------------------------------------------------------------
>
> Key: HADOOP-4033
> URL: https://issues.apache.org/jira/browse/HADOOP-4033
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.1
> Reporter: Runping Qi
> Assignee: Ravi Gummadi
>
> I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
> Consequently, it had hard time to finish the tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4033) Task tracker should not ask for
tasks if its temp disk space is almost full
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668964#action_12668964 ]
Arun C Murthy commented on HADOOP-4033:
---------------------------------------
bq. Actually, I'd propose that we backport the 0.19 items - HADOOP-657 (and probably another related issue - HADOOP-9) to handle this issue.
Uh, we can't add new features to a released branch?
> Task tracker should not ask for tasks if its temp disk space is almost full
> ---------------------------------------------------------------------------
>
> Key: HADOOP-4033
> URL: https://issues.apache.org/jira/browse/HADOOP-4033
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.1
> Reporter: Runping Qi
> Assignee: Ravi Gummadi
>
> I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
> Consequently, it had hard time to finish the tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4033) Task tracker should not ask for
tasks if its temp disk space is almost full
Posted by "Ravi Gummadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Gummadi reassigned HADOOP-4033:
------------------------------------
Assignee: Ravi Gummadi
> Task tracker should not ask for tasks if its temp disk space is almost full
> ---------------------------------------------------------------------------
>
> Key: HADOOP-4033
> URL: https://issues.apache.org/jira/browse/HADOOP-4033
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.1
> Reporter: Runping Qi
> Assignee: Ravi Gummadi
>
> I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
> Consequently, it had hard time to finish the tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4033) Task tracker should not ask for
tasks if its temp disk space is almost full
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668791#action_12668791 ]
Devaraj Das commented on HADOOP-4033:
-------------------------------------
Actually, I'd propose that we backport the 0.19 items - HADOOP-657 (and probably another related issue - HADOOP-9) to handle this issue.. HADOOP-657 seems like a better approach for the problem described.. thoughts?
> Task tracker should not ask for tasks if its temp disk space is almost full
> ---------------------------------------------------------------------------
>
> Key: HADOOP-4033
> URL: https://issues.apache.org/jira/browse/HADOOP-4033
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.1
> Reporter: Runping Qi
> Assignee: Ravi Gummadi
>
> I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
> Consequently, it had hard time to finish the tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4033) Task tracker should not ask for
tasks if its temp disk space is almost full
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12668793#action_12668793 ]
Devaraj Das commented on HADOOP-4033:
-------------------------------------
In addition to HADOOP-657, setting some sensible value for the minspacestart in mapred-site.xml should also be done
> Task tracker should not ask for tasks if its temp disk space is almost full
> ---------------------------------------------------------------------------
>
> Key: HADOOP-4033
> URL: https://issues.apache.org/jira/browse/HADOOP-4033
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.1
> Reporter: Runping Qi
> Assignee: Ravi Gummadi
>
> I observed a case where a task tracker still asked for task even though the available disk space on the machine is less than 1%.
> Consequently, it had hard time to finish the tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.