You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Leitao Guo (JIRA)" <ji...@apache.org> on 2009/03/18 08:19:50 UTC
[jira] Created: (HADOOP-5524) DiskChecker$DiskErrorException when
'reduce > reduce'
DiskChecker$DiskErrorException when 'reduce > reduce'
-----------------------------------------------------
Key: HADOOP-5524
URL: https://issues.apache.org/jira/browse/HADOOP-5524
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.19.1
Environment: hadoop-0.19.1; CentOS 5
Reporter: Leitao Guo
We have 9900 maptasks and 60 reducetasks in the job. When all the other 59 reducetasks have finished, the last reducetask runs so slow and finally finished after throwing out a lot of DiskErrorExceptions.
The following is the tasktracker log on which the reducetask is running.
2009-03-18 14:39:52,025 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:39:57,028 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:40:00,695 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200903171026_0091_r_000030_0 0.977961% reduce > reduce
2009-03-18 14:40:02,032 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:40:07,036 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:40:12,040 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:40:17,045 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:40:22,050 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:40:27,054 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:40:32,058 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:40:37,062 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:40:42,066 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:40:47,136 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:40:52,140 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:40:57,143 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:41:02,147 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:41:06,760 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200903171026_0091_r_000030_0 0.9788374% reduce > reduce
2009-03-18 14:41:07,152 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:41:09,762 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200903171026_0091_r_000030_0 0.9788374% reduce > reduce
2009-03-18 14:41:12,158 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:41:17,162 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:41:22,168 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
2009-03-18 14:41:27,172 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5524) DiskChecker$DiskErrorException when
'reduce > reduce'
Posted by "Chris K Wensel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12695817#action_12695817 ]
Chris K Wensel commented on HADOOP-5524:
----------------------------------------
Was this issue resolved?
> DiskChecker$DiskErrorException when 'reduce > reduce'
> -----------------------------------------------------
>
> Key: HADOOP-5524
> URL: https://issues.apache.org/jira/browse/HADOOP-5524
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.1
> Environment: hadoop-0.19.1; CentOS 5
> Reporter: Leitao Guo
>
> We have 9900 maptasks and 60 reducetasks in the job. When all the other 59 reducetasks have finished, the last reducetask runs so slow and finally finished after throwing out a lot of DiskErrorExceptions.
> The following is the tasktracker log on which the reducetask is running.
> 2009-03-18 14:39:52,025 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:39:57,028 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:00,695 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200903171026_0091_r_000030_0 0.977961% reduce > reduce
> 2009-03-18 14:40:02,032 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:07,036 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:12,040 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:17,045 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:22,050 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:27,054 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:32,058 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:37,062 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:42,066 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:47,136 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:52,140 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:57,143 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:41:02,147 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:41:06,760 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200903171026_0091_r_000030_0 0.9788374% reduce > reduce
> 2009-03-18 14:41:07,152 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:41:09,762 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200903171026_0091_r_000030_0 0.9788374% reduce > reduce
> 2009-03-18 14:41:12,158 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:41:17,162 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:41:22,168 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:41:27,172 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-5524)
DiskChecker$DiskErrorException when 'reduce > reduce'
Posted by "Chris K Wensel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12695817#action_12695817 ]
Chris K Wensel edited comment on HADOOP-5524 at 4/5/09 12:34 AM:
-----------------------------------------------------------------
Was this issue resolved? Looks related to HADOOP-4963.
was (Author: cwensel):
Was this issue resolved?
> DiskChecker$DiskErrorException when 'reduce > reduce'
> -----------------------------------------------------
>
> Key: HADOOP-5524
> URL: https://issues.apache.org/jira/browse/HADOOP-5524
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.1
> Environment: hadoop-0.19.1; CentOS 5
> Reporter: Leitao Guo
>
> We have 9900 maptasks and 60 reducetasks in the job. When all the other 59 reducetasks have finished, the last reducetask runs so slow and finally finished after throwing out a lot of DiskErrorExceptions.
> The following is the tasktracker log on which the reducetask is running.
> 2009-03-18 14:39:52,025 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:39:57,028 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:00,695 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200903171026_0091_r_000030_0 0.977961% reduce > reduce
> 2009-03-18 14:40:02,032 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:07,036 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:12,040 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:17,045 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:22,050 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:27,054 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:32,058 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:37,062 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:42,066 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:47,136 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:52,140 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:40:57,143 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:41:02,147 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:41:06,760 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200903171026_0091_r_000030_0 0.9788374% reduce > reduce
> 2009-03-18 14:41:07,152 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:41:09,762 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200903171026_0091_r_000030_0 0.9788374% reduce > reduce
> 2009-03-18 14:41:12,158 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:41:17,162 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:41:22,168 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
> 2009-03-18 14:41:27,172 INFO org.apache.hadoop.mapred.TaskTracker: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_200903171026_0091/attempt_200903171026_0091_r_000030_0/output/file.out in any of the configured local directories
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.