You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2006/12/12 05:10:21 UTC
[jira] Created: (HADOOP-813) map tasks lost during sort
map tasks lost during sort
--------------------------
Key: HADOOP-813
URL: http://issues.apache.org/jira/browse/HADOOP-813
Project: Hadoop
Issue Type: Bug
Components: mapred
Affects Versions: 0.10.0
Reporter: Owen O'Malley
Assigned To: Devaraj Das
Fix For: 0.10.0
On a 500 node cluster, I had a bunch of map tasks get "lost" because they failed to report progress for 10 minutes. They appear to be in the sort stage at the end of the map. I hypothesize that the patch for HADOOP-331 does not update the map's progress during the sort/merge. If the sort/merge takes more than 10 minutes, the task is lost.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-813) map tasks lost during sort
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-813?page=comments#action_12458626 ]
Owen O'Malley commented on HADOOP-813:
--------------------------------------
+1
> map tasks lost during sort
> --------------------------
>
> Key: HADOOP-813
> URL: http://issues.apache.org/jira/browse/HADOOP-813
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.0
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.10.0
>
> Attachments: 813.patch, 813.patch
>
>
> On a 500 node cluster, I had a bunch of map tasks get "lost" because they failed to report progress for 10 minutes. They appear to be in the sort stage at the end of the map. I hypothesize that the patch for HADOOP-331 does not update the map's progress during the sort/merge. If the sort/merge takes more than 10 minutes, the task is lost.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-813) map tasks lost during sort
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-813?page=comments#action_12457857 ]
Owen O'Malley commented on HADOOP-813:
--------------------------------------
1. sortComplete should be declared volatile.
2. The sortProgress thread should use LOG.info for reporting exceptions rather than System.out.println.
3. I'd probably add a sortProgress.interrupt() when you are closing down so that it can shut down faster.
4. The sortProgress thread should be a daemon thread.
5. There seem to be a lot of other changes in this patch that aren't related to the issue and aren't mentioned in your comment. Are they intentional?
> map tasks lost during sort
> --------------------------
>
> Key: HADOOP-813
> URL: http://issues.apache.org/jira/browse/HADOOP-813
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.0
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.10.0
>
> Attachments: 813.patch
>
>
> On a 500 node cluster, I had a bunch of map tasks get "lost" because they failed to report progress for 10 minutes. They appear to be in the sort stage at the end of the map. I hypothesize that the patch for HADOOP-331 does not update the map's progress during the sort/merge. If the sort/merge takes more than 10 minutes, the task is lost.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-813) map tasks lost during sort
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-813?page=comments#action_12457628 ]
Hadoop QA commented on HADOOP-813:
----------------------------------
+1, http://issues.apache.org/jira/secure/attachment/12346976/813.patch applied and successfully tested against trunk revision r483772
> map tasks lost during sort
> --------------------------
>
> Key: HADOOP-813
> URL: http://issues.apache.org/jira/browse/HADOOP-813
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.0
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.10.0
>
> Attachments: 813.patch
>
>
> On a 500 node cluster, I had a bunch of map tasks get "lost" because they failed to report progress for 10 minutes. They appear to be in the sort stage at the end of the map. I hypothesize that the patch for HADOOP-331 does not update the map's progress during the sort/merge. If the sort/merge takes more than 10 minutes, the task is lost.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-813) map tasks lost during sort
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-813?page=all ]
Devaraj Das updated HADOOP-813:
-------------------------------
Attachment: 813.patch
Thanks Owen for the review. I have removed the sortComplete variable and made the thread exit only on interrupt instead.
The comment on "lots of changes" is actually due to the fact that I introduced a try{ } finally{ } block and wrapped/indented existing code within the 'try' block. This led to lots of diffs.
> map tasks lost during sort
> --------------------------
>
> Key: HADOOP-813
> URL: http://issues.apache.org/jira/browse/HADOOP-813
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.0
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.10.0
>
> Attachments: 813.patch, 813.patch
>
>
> On a 500 node cluster, I had a bunch of map tasks get "lost" because they failed to report progress for 10 minutes. They appear to be in the sort stage at the end of the map. I hypothesize that the patch for HADOOP-331 does not update the map's progress during the sort/merge. If the sort/merge takes more than 10 minutes, the task is lost.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-813) map tasks lost during sort
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-813?page=all ]
Owen O'Malley updated HADOOP-813:
---------------------------------
Status: Open (was: Patch Available)
> map tasks lost during sort
> --------------------------
>
> Key: HADOOP-813
> URL: http://issues.apache.org/jira/browse/HADOOP-813
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.0
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.10.0
>
> Attachments: 813.patch
>
>
> On a 500 node cluster, I had a bunch of map tasks get "lost" because they failed to report progress for 10 minutes. They appear to be in the sort stage at the end of the map. I hypothesize that the patch for HADOOP-331 does not update the map's progress during the sort/merge. If the sort/merge takes more than 10 minutes, the task is lost.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-813) map tasks lost during sort
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-813?page=all ]
Devaraj Das updated HADOOP-813:
-------------------------------
Status: Patch Available (was: Open)
> map tasks lost during sort
> --------------------------
>
> Key: HADOOP-813
> URL: http://issues.apache.org/jira/browse/HADOOP-813
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.0
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.10.0
>
> Attachments: 813.patch
>
>
> On a 500 node cluster, I had a bunch of map tasks get "lost" because they failed to report progress for 10 minutes. They appear to be in the sort stage at the end of the map. I hypothesize that the patch for HADOOP-331 does not update the map's progress during the sort/merge. If the sort/merge takes more than 10 minutes, the task is lost.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HADOOP-813) map tasks lost during sort
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-813?page=all ]
Doug Cutting resolved HADOOP-813.
---------------------------------
Resolution: Fixed
I just committed this. Thanks, Devaraj!
> map tasks lost during sort
> --------------------------
>
> Key: HADOOP-813
> URL: http://issues.apache.org/jira/browse/HADOOP-813
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.0
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.10.0
>
> Attachments: 813.patch, 813.patch
>
>
> On a 500 node cluster, I had a bunch of map tasks get "lost" because they failed to report progress for 10 minutes. They appear to be in the sort stage at the end of the map. I hypothesize that the patch for HADOOP-331 does not update the map's progress during the sort/merge. If the sort/merge takes more than 10 minutes, the task is lost.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-813) map tasks lost during sort
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-813?page=all ]
Devaraj Das updated HADOOP-813:
-------------------------------
Attachment: 813.patch
This patch starts a thread to report progress during the merge process. Also, added a couple of reportProgress statements.
> map tasks lost during sort
> --------------------------
>
> Key: HADOOP-813
> URL: http://issues.apache.org/jira/browse/HADOOP-813
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.10.0
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Fix For: 0.10.0
>
> Attachments: 813.patch
>
>
> On a 500 node cluster, I had a bunch of map tasks get "lost" because they failed to report progress for 10 minutes. They appear to be in the sort stage at the end of the map. I hypothesize that the patch for HADOOP-331 does not update the map's progress during the sort/merge. If the sort/merge takes more than 10 minutes, the task is lost.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira