You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2007/06/21 10:32:25 UTC
[jira] Created: (HADOOP-1514) Progress reporting not handled for
the case of empty map outputs
Progress reporting not handled for the case of empty map outputs
----------------------------------------------------------------
Key: HADOOP-1514
URL: https://issues.apache.org/jira/browse/HADOOP-1514
Project: Hadoop
Issue Type: Bug
Components: mapred
Affects Versions: 0.14.0
Reporter: Devaraj Das
Assignee: Vivek Ratan
Priority: Blocker
Fix For: 0.14.0
For the randomwriter run, the maps actually don't generate any output, and the reducer is just waiting for the maps to complete. During that time, progress reporting is not done and hence the process gets killed. Also the case of empty map outputs needs to be handled more generally.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1514) Progress reporting not handled for
the case where a reducer currently doesn't have anything to fetch
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12508510 ]
Hudson commented on HADOOP-1514:
--------------------------------
Integrated in Hadoop-Nightly #137 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/137/])
> Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1514
> URL: https://issues.apache.org/jira/browse/HADOOP-1514
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Devaraj Das
> Assignee: Vivek Ratan
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: 1514_01.patch
>
>
> In some apps (like the randomwriter), the maps take a long time to complete, and there could be cases where a reducer would just be waiting for map(s) to complete. During that time progress reporting is not done.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1514) Progress reporting not handled for
the case where a reducer currently doesn't have anything to fetch
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-1514:
--------------------------------
Description: In some apps (like the randomwriter), the maps take a long time to complete, and there could be cases where a reducer would just be waiting for map(s) to complete. During that time progress reporting is not done. (was: For the randomwriter run, the maps actually don't generate any output, and the reducer is just waiting for the maps to complete. During that time, progress reporting is not done and hence the process gets killed. Also the case of empty map outputs needs to be handled more generally.)
Summary: Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch (was: Progress reporting not handled for the case of empty map outputs)
> Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1514
> URL: https://issues.apache.org/jira/browse/HADOOP-1514
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Devaraj Das
> Assignee: Vivek Ratan
> Priority: Blocker
> Fix For: 0.14.0
>
>
> In some apps (like the randomwriter), the maps take a long time to complete, and there could be cases where a reducer would just be waiting for map(s) to complete. During that time progress reporting is not done.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1514) Progress reporting not handled for
the case where a reducer currently doesn't have anything to fetch
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-1514:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Vivek!
> Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1514
> URL: https://issues.apache.org/jira/browse/HADOOP-1514
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Devaraj Das
> Assignee: Vivek Ratan
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: 1514_01.patch
>
>
> In some apps (like the randomwriter), the maps take a long time to complete, and there could be cases where a reducer would just be waiting for map(s) to complete. During that time progress reporting is not done.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1514) Progress reporting not handled for
the case where a reducer currently doesn't have anything to fetch
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12508067 ]
Devaraj Das commented on HADOOP-1514:
-------------------------------------
+1
> Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1514
> URL: https://issues.apache.org/jira/browse/HADOOP-1514
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Devaraj Das
> Assignee: Vivek Ratan
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: 1514_01.patch
>
>
> In some apps (like the randomwriter), the maps take a long time to complete, and there could be cases where a reducer would just be waiting for map(s) to complete. During that time progress reporting is not done.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1514) Progress reporting not handled for
the case where a reducer currently doesn't have anything to fetch
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12508078 ]
Hadoop QA commented on HADOOP-1514:
-----------------------------------
+1
http://issues.apache.org/jira/secure/attachment/12360538/1514_01.patch applied and successfully tested against trunk revision r550635.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/331/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/331/console
> Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1514
> URL: https://issues.apache.org/jira/browse/HADOOP-1514
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Devaraj Das
> Assignee: Vivek Ratan
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: 1514_01.patch
>
>
> In some apps (like the randomwriter), the maps take a long time to complete, and there could be cases where a reducer would just be waiting for map(s) to complete. During that time progress reporting is not done.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1514) Progress reporting not handled for
the case where a reducer currently doesn't have anything to fetch
Posted by "Vivek Ratan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vivek Ratan updated HADOOP-1514:
--------------------------------
Attachment: 1514_01.patch
The fix is to report progress in the Reducer if there are no downloads of map output in progress or none scheduled. See 1514_01.patch.
> Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1514
> URL: https://issues.apache.org/jira/browse/HADOOP-1514
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Devaraj Das
> Assignee: Vivek Ratan
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: 1514_01.patch
>
>
> In some apps (like the randomwriter), the maps take a long time to complete, and there could be cases where a reducer would just be waiting for map(s) to complete. During that time progress reporting is not done.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1514) Progress reporting not handled for
the case where a reducer currently doesn't have anything to fetch
Posted by "Vivek Ratan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vivek Ratan updated HADOOP-1514:
--------------------------------
Status: Patch Available (was: Open)
> Progress reporting not handled for the case where a reducer currently doesn't have anything to fetch
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-1514
> URL: https://issues.apache.org/jira/browse/HADOOP-1514
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0
> Reporter: Devaraj Das
> Assignee: Vivek Ratan
> Priority: Blocker
> Fix For: 0.14.0
>
> Attachments: 1514_01.patch
>
>
> In some apps (like the randomwriter), the maps take a long time to complete, and there could be cases where a reducer would just be waiting for map(s) to complete. During that time progress reporting is not done.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.