You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org> on 2009/04/21 06:36:47 UTC
[jira] Created: (HADOOP-5710) Counter MAP_INPUT_BYTES missing from
new mapreduce api.
Counter MAP_INPUT_BYTES missing from new mapreduce api.
-------------------------------------------------------
Key: HADOOP-5710
URL: https://issues.apache.org/jira/browse/HADOOP-5710
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Reporter: Amareshwari Sriramadasu
Assignee: Amareshwari Sriramadasu
Fix For: 0.21.0
MapTask, with current new mapreduce api, does not maintain MAP_INPUT_BYTES counter. Since RecordReader doesnot have getPos() api, it is not possible for the map task to maintain. Individual record readers (LineRecordReader, SequenceFileRecordReader) could be modified to maintain the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5710) Counter MAP_INPUT_BYTES missing from
new mapreduce api.
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5710:
--------------------------------------------
Attachment: patch-5710.txt
Patch adding MAP_INPUT_BYTES counter to LineRecordReader and SequenceRecordReader.
> Counter MAP_INPUT_BYTES missing from new mapreduce api.
> -------------------------------------------------------
>
> Key: HADOOP-5710
> URL: https://issues.apache.org/jira/browse/HADOOP-5710
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: patch-5710.txt
>
>
> MapTask, with current new mapreduce api, does not maintain MAP_INPUT_BYTES counter. Since RecordReader doesnot have getPos() api, it is not possible for the map task to maintain. Individual record readers (LineRecordReader, SequenceFileRecordReader) could be modified to maintain the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5710) Counter MAP_INPUT_BYTES missing from
new mapreduce api.
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5710:
--------------------------------------------
Attachment: patch-5710-3.txt
Patch incorporating review comments.
test-patch result:
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 12 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
{noformat}
ant test passed on my machine.
> Counter MAP_INPUT_BYTES missing from new mapreduce api.
> -------------------------------------------------------
>
> Key: HADOOP-5710
> URL: https://issues.apache.org/jira/browse/HADOOP-5710
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: patch-5710-1.txt, patch-5710-2.txt, patch-5710-3.txt, patch-5710.txt
>
>
> MapTask, with current new mapreduce api, does not maintain MAP_INPUT_BYTES counter. Since RecordReader doesnot have getPos() api, it is not possible for the map task to maintain. Individual record readers (LineRecordReader, SequenceFileRecordReader) could be modified to maintain the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5710) Counter MAP_INPUT_BYTES missing from
new mapreduce api.
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5710:
--------------------------------------------
Attachment: patch-5710-2.txt
Patch incorporating review comments. Earlier patch had findbugs warning saying Unchecked/unconfirmed cast fromTaskAttemptContext to MapContext . Added the warning to exclude list, since the warning is unavoidable.
> Counter MAP_INPUT_BYTES missing from new mapreduce api.
> -------------------------------------------------------
>
> Key: HADOOP-5710
> URL: https://issues.apache.org/jira/browse/HADOOP-5710
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: patch-5710-1.txt, patch-5710-2.txt, patch-5710.txt
>
>
> MapTask, with current new mapreduce api, does not maintain MAP_INPUT_BYTES counter. Since RecordReader doesnot have getPos() api, it is not possible for the map task to maintain. Individual record readers (LineRecordReader, SequenceFileRecordReader) could be modified to maintain the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5710) Counter MAP_INPUT_BYTES missing
from new mapreduce api.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710644#action_12710644 ]
Owen O'Malley commented on HADOOP-5710:
---------------------------------------
That sounds reasonable, Sharad.
> Counter MAP_INPUT_BYTES missing from new mapreduce api.
> -------------------------------------------------------
>
> Key: HADOOP-5710
> URL: https://issues.apache.org/jira/browse/HADOOP-5710
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: patch-5710.txt
>
>
> MapTask, with current new mapreduce api, does not maintain MAP_INPUT_BYTES counter. Since RecordReader doesnot have getPos() api, it is not possible for the map task to maintain. Individual record readers (LineRecordReader, SequenceFileRecordReader) could be modified to maintain the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5710) Counter MAP_INPUT_BYTES missing from
new mapreduce api.
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5710:
--------------------------------------------
Attachment: patch-5710-1.txt
Attaching patch which moves MAP_INPUT_BYTES to org.apache.hadoop.mapreduce.lib.input.FileInputFormat
> Counter MAP_INPUT_BYTES missing from new mapreduce api.
> -------------------------------------------------------
>
> Key: HADOOP-5710
> URL: https://issues.apache.org/jira/browse/HADOOP-5710
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: patch-5710-1.txt, patch-5710.txt
>
>
> MapTask, with current new mapreduce api, does not maintain MAP_INPUT_BYTES counter. Since RecordReader doesnot have getPos() api, it is not possible for the map task to maintain. Individual record readers (LineRecordReader, SequenceFileRecordReader) could be modified to maintain the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5710) Counter MAP_INPUT_BYTES missing
from new mapreduce api.
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712628#action_12712628 ]
Amareshwari Sriramadasu commented on HADOOP-5710:
-------------------------------------------------
Contrib test failures are not related to patch. They fail on trunk also.
> Counter MAP_INPUT_BYTES missing from new mapreduce api.
> -------------------------------------------------------
>
> Key: HADOOP-5710
> URL: https://issues.apache.org/jira/browse/HADOOP-5710
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: patch-5710-1.txt, patch-5710-2.txt, patch-5710.txt
>
>
> MapTask, with current new mapreduce api, does not maintain MAP_INPUT_BYTES counter. Since RecordReader doesnot have getPos() api, it is not possible for the map task to maintain. Individual record readers (LineRecordReader, SequenceFileRecordReader) could be modified to maintain the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5710) Counter MAP_INPUT_BYTES missing from
new mapreduce api.
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5710:
--------------------------------------------
Status: Patch Available (was: Open)
test-patch result :
{noformat}
[exec]
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 12 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
{noformat}
ant test passed on my machine.
> Counter MAP_INPUT_BYTES missing from new mapreduce api.
> -------------------------------------------------------
>
> Key: HADOOP-5710
> URL: https://issues.apache.org/jira/browse/HADOOP-5710
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: patch-5710-1.txt, patch-5710-2.txt, patch-5710.txt
>
>
> MapTask, with current new mapreduce api, does not maintain MAP_INPUT_BYTES counter. Since RecordReader doesnot have getPos() api, it is not possible for the map task to maintain. Individual record readers (LineRecordReader, SequenceFileRecordReader) could be modified to maintain the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5710) Counter MAP_INPUT_BYTES missing
from new mapreduce api.
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711476#action_12711476 ]
Sharad Agarwal commented on HADOOP-5710:
----------------------------------------
Patch looks good. Few minor points:
should we rename the group and counter to more readable and intuitive value. Say Group name as "FileInputFormatCounters" and counter name as "BYTES_READ", Because anyway we are calculating the bytes read from the file, not really the bytes input to the map (there may be some bytes skipped while reading so these two may not be same).
include a deprecated warning in the old Counters API while mapping the old name to new name.
> Counter MAP_INPUT_BYTES missing from new mapreduce api.
> -------------------------------------------------------
>
> Key: HADOOP-5710
> URL: https://issues.apache.org/jira/browse/HADOOP-5710
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: patch-5710-1.txt, patch-5710.txt
>
>
> MapTask, with current new mapreduce api, does not maintain MAP_INPUT_BYTES counter. Since RecordReader doesnot have getPos() api, it is not possible for the map task to maintain. Individual record readers (LineRecordReader, SequenceFileRecordReader) could be modified to maintain the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5710) Counter MAP_INPUT_BYTES missing
from new mapreduce api.
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712658#action_12712658 ]
Sharad Agarwal commented on HADOOP-5710:
----------------------------------------
one minor bit: In LineRecordReader and SequenceFileRecordReader, counter lookup is happening for each key. This can be avoided if we keep counter instance as the member variable as opposed to keeping MapContext.
> Counter MAP_INPUT_BYTES missing from new mapreduce api.
> -------------------------------------------------------
>
> Key: HADOOP-5710
> URL: https://issues.apache.org/jira/browse/HADOOP-5710
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: patch-5710-1.txt, patch-5710-2.txt, patch-5710.txt
>
>
> MapTask, with current new mapreduce api, does not maintain MAP_INPUT_BYTES counter. Since RecordReader doesnot have getPos() api, it is not possible for the map task to maintain. Individual record readers (LineRecordReader, SequenceFileRecordReader) could be modified to maintain the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5710) Counter MAP_INPUT_BYTES missing
from new mapreduce api.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712430#action_12712430 ]
Hadoop QA commented on HADOOP-5710:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12408678/patch-5710-2.txt
against trunk revision 777761.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/387/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/387/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/387/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/387/console
This message is automatically generated.
> Counter MAP_INPUT_BYTES missing from new mapreduce api.
> -------------------------------------------------------
>
> Key: HADOOP-5710
> URL: https://issues.apache.org/jira/browse/HADOOP-5710
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: patch-5710-1.txt, patch-5710-2.txt, patch-5710.txt
>
>
> MapTask, with current new mapreduce api, does not maintain MAP_INPUT_BYTES counter. Since RecordReader doesnot have getPos() api, it is not possible for the map task to maintain. Individual record readers (LineRecordReader, SequenceFileRecordReader) could be modified to maintain the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5710) Counter MAP_INPUT_BYTES missing from
new mapreduce api.
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5710:
--------------------------------------------
Status: Open (was: Patch Available)
> Counter MAP_INPUT_BYTES missing from new mapreduce api.
> -------------------------------------------------------
>
> Key: HADOOP-5710
> URL: https://issues.apache.org/jira/browse/HADOOP-5710
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: patch-5710.txt
>
>
> MapTask, with current new mapreduce api, does not maintain MAP_INPUT_BYTES counter. Since RecordReader doesnot have getPos() api, it is not possible for the map task to maintain. Individual record readers (LineRecordReader, SequenceFileRecordReader) could be modified to maintain the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5710) Counter MAP_INPUT_BYTES missing from
new mapreduce api.
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-5710:
-----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks Amareshwari!
> Counter MAP_INPUT_BYTES missing from new mapreduce api.
> -------------------------------------------------------
>
> Key: HADOOP-5710
> URL: https://issues.apache.org/jira/browse/HADOOP-5710
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: patch-5710-1.txt, patch-5710-2.txt, patch-5710-3.txt, patch-5710.txt
>
>
> MapTask, with current new mapreduce api, does not maintain MAP_INPUT_BYTES counter. Since RecordReader doesnot have getPos() api, it is not possible for the map task to maintain. Individual record readers (LineRecordReader, SequenceFileRecordReader) could be modified to maintain the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5710) Counter MAP_INPUT_BYTES missing from
new mapreduce api.
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5710:
--------------------------------------------
Status: Patch Available (was: Open)
Please apply the patch after commiting HADOOP-5717, because compilation would break.
> Counter MAP_INPUT_BYTES missing from new mapreduce api.
> -------------------------------------------------------
>
> Key: HADOOP-5710
> URL: https://issues.apache.org/jira/browse/HADOOP-5710
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: patch-5710.txt
>
>
> MapTask, with current new mapreduce api, does not maintain MAP_INPUT_BYTES counter. Since RecordReader doesnot have getPos() api, it is not possible for the map task to maintain. Individual record readers (LineRecordReader, SequenceFileRecordReader) could be modified to maintain the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5710) Counter MAP_INPUT_BYTES missing
from new mapreduce api.
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705915#action_12705915 ]
Sharad Agarwal commented on HADOOP-5710:
----------------------------------------
I propose that we remove MAP_INPUT_BYTES from the framework counters as it is specific to file based input formats only. With new api, the framework no more calculates this. It is getting calculated in lib. So we can move MAP_INPUT_BYTES to FileInputFormat class. That seems to be its more logical place.
> Counter MAP_INPUT_BYTES missing from new mapreduce api.
> -------------------------------------------------------
>
> Key: HADOOP-5710
> URL: https://issues.apache.org/jira/browse/HADOOP-5710
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.21.0
>
> Attachments: patch-5710.txt
>
>
> MapTask, with current new mapreduce api, does not maintain MAP_INPUT_BYTES counter. Since RecordReader doesnot have getPos() api, it is not possible for the map task to maintain. Individual record readers (LineRecordReader, SequenceFileRecordReader) could be modified to maintain the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.