You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Jothi Padmanabhan (JIRA)" <ji...@apache.org> on 2009/05/21 08:20:45 UTC
[jira] Created: (HADOOP-5882) Progress is not updated when the New
Reducer is running reduce phase
Progress is not updated when the New Reducer is running reduce phase
--------------------------------------------------------------------
Key: HADOOP-5882
URL: https://issues.apache.org/jira/browse/HADOOP-5882
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Reporter: Jothi Padmanabhan
The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5882) Progress is not updated when the New
Reducer is running reduce phase
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5882:
--------------------------------------------
Assignee: Amareshwari Sriramadasu
Status: Patch Available (was: Open)
test-patch result:
{noformat}
[exec]
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] -1 release audit. The applied patch generated 497 release audit warnings (more than the trunk's current 495 warnings).
[exec]
{noformat}
-1 release audit. There are no new files added in the patch. I dont know why it is giving -1 release audit.
-1 tests included. It is difficult to a unit test for observing progress updates during reduce phase. Patch has been tested manually
All mapred unit tests passed on my machine.
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5882) Progress is not updated when the New
Reducer is running reduce phase
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5882:
--------------------------------------------
Attachment: patch-5882.txt
Patch adding progress during reduce phase. Tested the patch manually to see progress updates during reduce phase.
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5882) Progress is not updated when the New
Reducer is running reduce phase
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated HADOOP-5882:
--------------------------------------
Fix Version/s: 0.21.0
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5882) Progress is not updated when the
New Reducer is running reduce phase
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711959#action_12711959 ]
Owen O'Malley commented on HADOOP-5882:
---------------------------------------
This needs to be addressed like the MapTask does it where the progress is updated when the code calls next. In the new api, the loop is handled in user code and therefore can't require the extra call to update progress.
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Priority: Blocker
> Fix For: 0.20.1
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5882) Progress is not updated when the
New Reducer is running reduce phase
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715357#action_12715357 ]
Amareshwari Sriramadasu commented on HADOOP-5882:
-------------------------------------------------
bq.Seems that this was committed to 0.20 only but not 0.21.
It is committed to 0.21, but not 0.20.1
The patch has been verified by running sort with new api, and observing progress update in redude from 66% to 100% (the reduce phase).
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882-1.txt, patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5882) Progress is not updated when the New
Reducer is running reduce phase
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5882:
--------------------------------------------
Attachment: patch-5882-0.20.txt
Patch for branch 0.20.1
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882-0.20.txt, patch-5882-1.txt, patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5882) Progress is not updated when the
New Reducer is running reduce phase
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712953#action_12712953 ]
Amareshwari Sriramadasu commented on HADOOP-5882:
-------------------------------------------------
Test failures are not related to the patch.
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5882) Progress is not updated when the
New Reducer is running reduce phase
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712828#action_12712828 ]
Hadoop QA commented on HADOOP-5882:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12408794/patch-5882.txt
against trunk revision 778388.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
-1 release audit. The applied patch generated 493 release audit warnings (more than the trunk's current 491 warnings).
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/398/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/398/artifact/trunk/current/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/398/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/398/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/398/console
This message is automatically generated.
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5882) Progress is not updated when the New
Reducer is running reduce phase
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5882:
--------------------------------------------
Status: Patch Available (was: Open)
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882-0.20.txt, patch-5882-1.txt, patch-5882-2.txt, patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5882) Progress is not updated when the New
Reducer is running reduce phase
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5882:
--------------------------------------------
Attachment: patch-5882-1.txt
Code changes as suggested Sharad, wraping RawKeyValueIterator to report progress. Tested the patch on cluster, it works fine
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882-1.txt, patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5882) Progress is not updated when the
New Reducer is running reduce phase
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715798#action_12715798 ]
Amareshwari Sriramadasu commented on HADOOP-5882:
-------------------------------------------------
ant test and test patch passed on my machine.
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882-0.20.txt, patch-5882-1.txt, patch-5882-2.txt, patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5882) Progress is not updated when the
New Reducer is running reduce phase
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715421#action_12715421 ]
Amareshwari Sriramadasu commented on HADOOP-5882:
-------------------------------------------------
Patch does not apply to 0.20. I'm testing the patch for 0.20. Will upload a patch soon.
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882-1.txt, patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5882) Progress is not updated when the New
Reducer is running reduce phase
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5882:
--------------------------------------------
Status: Open (was: Patch Available)
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5882) Progress is not updated when the
New Reducer is running reduce phase
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12718618#action_12718618 ]
Hudson commented on HADOOP-5882:
--------------------------------
Integrated in Hadoop-trunk #863 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/863/])
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1
>
> Attachments: patch-5882-0.20.txt, patch-5882-1.txt, patch-5882-2.txt, patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5882) Progress is not updated when the New
Reducer is running reduce phase
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5882:
--------------------------------------------
Priority: Blocker (was: Major)
Affects Version/s: 0.20.0
Fix Version/s: 0.20.1
Marking it a blocker, I see reducers failing because they couldn't report status for 600 seconds, during reduce phase.
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Priority: Blocker
> Fix For: 0.20.1
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5882) Progress is not updated when the
New Reducer is running reduce phase
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715407#action_12715407 ]
Amareshwari Sriramadasu commented on HADOOP-5882:
-------------------------------------------------
Patch does not apply to 0.20. I'm testing the patch for 0.20. Will upload a patch soon.
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882-1.txt, patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5882) Progress is not updated when the New
Reducer is running reduce phase
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-5882:
-----------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.21.0)
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I committed this to trunk and branch 0.20.1. Thanks Amareshwari!
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1
>
> Attachments: patch-5882-0.20.txt, patch-5882-1.txt, patch-5882-2.txt, patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5882) Progress is not updated when the
New Reducer is running reduce phase
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715478#action_12715478 ]
Amareshwari Sriramadasu commented on HADOOP-5882:
-------------------------------------------------
Uploaded patch doesnt solve the issue. Patch was wrongly created. I will upload correct patch for trunk and 0.20.
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882-1.txt, patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5882) Progress is not updated when the New
Reducer is running reduce phase
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5882:
--------------------------------------------
Attachment: patch-5882-2.txt
patch for trunk
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882-0.20.txt, patch-5882-1.txt, patch-5882-2.txt, patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5882) Progress is not updated when the
New Reducer is running reduce phase
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714799#action_12714799 ]
Hadoop QA commented on HADOOP-5882:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12409363/patch-5882-1.txt
against trunk revision 780114.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/437/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/437/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/437/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/437/console
This message is automatically generated.
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882-1.txt, patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5882) Progress is not updated when the
New Reducer is running reduce phase
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715187#action_12715187 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-5882:
------------------------------------------------
Seems that this was committed to 0.20 only but not 0.21.
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882-1.txt, patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5882) Progress is not updated when the New
Reducer is running reduce phase
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5882:
--------------------------------------------
Status: Patch Available (was: Open)
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882-1.txt, patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5882) Progress is not updated when the
New Reducer is running reduce phase
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714958#action_12714958 ]
Amareshwari Sriramadasu commented on HADOOP-5882:
-------------------------------------------------
Test failures are not related to the patch.
All tests passed on my machine
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882-1.txt, patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5882) Progress is not updated when the
New Reducer is running reduce phase
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714353#action_12714353 ]
Sharad Agarwal commented on HADOOP-5882:
----------------------------------------
Changing the ReduceContext and having the constructor with null progress object does not seem very elegant. Would it work, if we just intercept RawKeyValueIterator.next()? Something like:
{code}
final RawKeyValueIterator rawIter = rIter;
rIter = new RawKeyValueIterator() {
public void close() throws IOException {
rawIter.close();
}
public DataInputBuffer getKey() throws IOException {
return rawIter.getKey();
}
public Progress getProgress() {
return rawIter.getProgress();
}
public DataInputBuffer getValue() throws IOException {
return rawIter.getValue();
}
public boolean next() throws IOException {
reducePhase.set(rawIter.getProgress().getProgress());
return rawIter.next();
}
};
{code}
This looks much simpler to me.
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5882) Progress is not updated when the New
Reducer is running reduce phase
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-5882:
-----------------------------------
Status: Open (was: Patch Available)
> Progress is not updated when the New Reducer is running reduce phase
> --------------------------------------------------------------------
>
> Key: HADOOP-5882
> URL: https://issues.apache.org/jira/browse/HADOOP-5882
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Jothi Padmanabhan
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.20.1, 0.21.0
>
> Attachments: patch-5882-1.txt, patch-5882.txt
>
>
> The old reducer calls informReduceProgress on a key change to update the progress. This call or equivalent should be called for the new reducer as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.