You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Chris Douglas (JIRA)" <ji...@apache.org> on 2009/10/23 22:43:59 UTC
[jira] Created: (MAPREDUCE-1147) Map output records counter missing
for map-only jobs in new API
Map output records counter missing for map-only jobs in new API
---------------------------------------------------------------
Key: MAPREDUCE-1147
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.20.1, 0.21.0
Reporter: Chris Douglas
Priority: Blocker
Fix For: 0.20.2
In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1147) Map output records counter missing
for map-only jobs in new API
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated MAPREDUCE-1147:
----------------------------------
Attachment: mapred-1147-v1.3.patch
Attaching a patch that adds map-output-records to the map-only jobs submitted via new api. Test-patch o/p
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 7 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
All tests passed except TestGridmixSubmission. Debugging further.
> Map output records counter missing for map-only jobs in new API
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1147
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0
> Reporter: Chris Douglas
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapred-1147-v1.3.patch
>
>
> In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1147) Map output records counter missing
for map-only jobs in new API
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated MAPREDUCE-1147:
----------------------------------
Status: Patch Available (was: Open)
Test patch passed on my box.
Running through hudson.
> Map output records counter missing for map-only jobs in new API
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1147
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0
> Reporter: Chris Douglas
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapred-1147-v1.3.patch, mapred-1147-v1.4.patch
>
>
> In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1147) Map output records counter missing
for map-only jobs in new API
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated MAPREDUCE-1147:
----------------------------------
Attachment: mapred-1147-v1.4-y20.patch
Attaching a patch for branch 20. Result of test-patch
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 6 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
Testing now.
> Map output records counter missing for map-only jobs in new API
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1147
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0
> Reporter: Chris Douglas
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapred-1147-v1.3.patch, mapred-1147-v1.4-y20.patch, mapred-1147-v1.4.patch
>
>
> In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1147) Map output records counter missing
for map-only jobs in new API
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated MAPREDUCE-1147:
----------------------------------
Release Note: Adds a counter to track the number of records emitted by map writing directly to HDFS i.e map tasks of job with 0 reducers.
> Map output records counter missing for map-only jobs in new API
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1147
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0
> Reporter: Chris Douglas
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapred-1147-v1.3.patch, mapred-1147-v1.4-y20.patch, mapred-1147-v1.4.patch
>
>
> In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1147) Map output records counter
missing for map-only jobs in new API
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774584#action_12774584 ]
Amar Kamat commented on MAPREDUCE-1147:
---------------------------------------
TestGridmixSubmission fails intermittently.
> Map output records counter missing for map-only jobs in new API
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1147
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0
> Reporter: Chris Douglas
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapred-1147-v1.3.patch
>
>
> In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1147) Map output records counter
missing for map-only jobs in new API
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778696#action_12778696 ]
Hudson commented on MAPREDUCE-1147:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #120 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/120/])
. Add map output counters to new API. Contributed by Amar Kamat
> Map output records counter missing for map-only jobs in new API
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1147
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0
> Reporter: Chris Douglas
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapred-1147-v1.3.patch, mapred-1147-v1.4-y20.patch, mapred-1147-v1.4.patch
>
>
> In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1147) Map output records counter
missing for map-only jobs in new API
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778235#action_12778235 ]
Amar Kamat commented on MAPREDUCE-1147:
---------------------------------------
For branch 20, all tests except the ones mentioned below have passed.
# hdfs.TestDatanodeBlockScanner FAILED (timeout)
# hdfs.TestDistributedFileSystem FAILED
# hdfs.server.namenode.TestFsck FAILED (timeout)
# mapred.TestReduceFetch FAILED
None of these seems related to this issue.
> Map output records counter missing for map-only jobs in new API
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1147
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0
> Reporter: Chris Douglas
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapred-1147-v1.3.patch, mapred-1147-v1.4-y20.patch, mapred-1147-v1.4.patch
>
>
> In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1147) Map output records counter missing
for map-only jobs in new API
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1147:
-------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I committed this. Thanks, Amar!
> Map output records counter missing for map-only jobs in new API
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1147
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0
> Reporter: Chris Douglas
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapred-1147-v1.3.patch, mapred-1147-v1.4-y20.patch, mapred-1147-v1.4.patch
>
>
> In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1147) Map output records counter
missing for map-only jobs in new API
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774961#action_12774961 ]
Amar Kamat commented on MAPREDUCE-1147:
---------------------------------------
TestGridMixSubmission failure is a known issue, see MAPREDUCE-1124.
> Map output records counter missing for map-only jobs in new API
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1147
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0
> Reporter: Chris Douglas
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapred-1147-v1.3.patch
>
>
> In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1147) Map output records counter
missing for map-only jobs in new API
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776433#action_12776433 ]
Hadoop QA commented on MAPREDUCE-1147:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12424579/mapred-1147-v1.4.patch
against trunk revision 834284.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/console
This message is automatically generated.
> Map output records counter missing for map-only jobs in new API
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1147
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0
> Reporter: Chris Douglas
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapred-1147-v1.3.patch, mapred-1147-v1.4.patch
>
>
> In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1147) Map output records counter
missing for map-only jobs in new API
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779508#action_12779508 ]
Hudson commented on MAPREDUCE-1147:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #147 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/147/])
> Map output records counter missing for map-only jobs in new API
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1147
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0
> Reporter: Chris Douglas
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapred-1147-v1.3.patch, mapred-1147-v1.4-y20.patch, mapred-1147-v1.4.patch
>
>
> In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1147) Map output records counter
missing for map-only jobs in new API
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776900#action_12776900 ]
Chris Douglas commented on MAPREDUCE-1147:
------------------------------------------
+1 This looks good; would you mind attaching a patch for 0.20?
> Map output records counter missing for map-only jobs in new API
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1147
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0
> Reporter: Chris Douglas
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapred-1147-v1.3.patch, mapred-1147-v1.4.patch
>
>
> In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1147) Map output records counter missing
for map-only jobs in new API
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated MAPREDUCE-1147:
----------------------------------
Attachment: mapred-1147-v1.4.patch
Attaching a new patch that incorporates comments by Chris. Testing in progress.
> Map output records counter missing for map-only jobs in new API
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1147
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0
> Reporter: Chris Douglas
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapred-1147-v1.3.patch, mapred-1147-v1.4.patch
>
>
> In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1147) Map output records counter
missing for map-only jobs in new API
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat reassigned MAPREDUCE-1147:
-------------------------------------
Assignee: Amar Kamat
> Map output records counter missing for map-only jobs in new API
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1147
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0
> Reporter: Chris Douglas
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.20.2
>
>
> In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1147) Map output records counter
missing for map-only jobs in new API
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778274#action_12778274 ]
Amar Kamat commented on MAPREDUCE-1147:
---------------------------------------
Also for branch 20, all contrib tests passed except TestHdfsProxy and TestStreamingExitStatus.
> Map output records counter missing for map-only jobs in new API
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1147
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0
> Reporter: Chris Douglas
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapred-1147-v1.3.patch, mapred-1147-v1.4-y20.patch, mapred-1147-v1.4.patch
>
>
> In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1147) Map output records counter
missing for map-only jobs in new API
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776349#action_12776349 ]
Chris Douglas commented on MAPREDUCE-1147:
------------------------------------------
Looks good, though {{NewDirectOutputCollector::reporter}} can be final and the testcase should use the JUnit4 annotations instead of extending TestCase
> Map output records counter missing for map-only jobs in new API
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-1147
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.21.0
> Reporter: Chris Douglas
> Assignee: Amar Kamat
> Priority: Blocker
> Fix For: 0.20.2
>
> Attachments: mapred-1147-v1.3.patch
>
>
> In the new API, the counter for map output records is not incremented for map-only jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.