You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Chris Douglas (JIRA)" <ji...@apache.org> on 2009/10/06 07:19:31 UTC
[jira] Created: (MAPREDUCE-1061) Gridmix unit test should validate
input/output bytes
Gridmix unit test should validate input/output bytes
----------------------------------------------------
Key: MAPREDUCE-1061
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
Project: Hadoop Map/Reduce
Issue Type: Test
Affects Versions: 0.21.0
Reporter: Chris Douglas
Fix For: 0.21.0
TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1061) Gridmix unit test should validate
input/output bytes
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1061:
-------------------------------------
Status: Patch Available (was: Open)
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch, M1061-1.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1061) Gridmix unit test should
validate input/output bytes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767081#action_12767081 ]
Hudson commented on MAPREDUCE-1061:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #116 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/116/])
. Add unit test validating byte specifications for gridmix jobs.
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch, M1061-1.patch, M1061-2.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1061) Gridmix unit test should
validate input/output bytes
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764284#action_12764284 ]
Hong Tang commented on MAPREDUCE-1061:
--------------------------------------
Patch looks good. The detailed verification is rigorous, which is nice.
One minor nit: why do we need to set the actual extra bytes and records proportional to nMaps and nReds. Does it make sense that for map output, each map would output at most extra max(1, nReduce/nMaps) records? And for each reducer only one extra record?
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch, M1061-1.patch, M1061-2.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1061) Gridmix unit test should validate
input/output bytes
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas reassigned MAPREDUCE-1061:
----------------------------------------
Assignee: Chris Douglas
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1061) Gridmix unit test should
validate input/output bytes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763747#action_12763747 ]
Hadoop QA commented on MAPREDUCE-1061:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12421657/M1061-2.patch
against trunk revision 823227.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/153/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/153/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/153/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/153/console
This message is automatically generated.
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch, M1061-1.patch, M1061-2.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1061) Gridmix unit test should
validate input/output bytes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763651#action_12763651 ]
Hadoop QA commented on MAPREDUCE-1061:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12421634/M1061-1.patch
against trunk revision 823227.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/152/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/152/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/152/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/152/console
This message is automatically generated.
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch, M1061-1.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1061) Gridmix unit test should validate
input/output bytes
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1061:
-------------------------------------
Attachment: 1061-0.patch
Verifies that input/output bytes are within 32k, records within 5 for each task (spec bytes for reduce tasks currently introduce extra intermediate data)
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1061) Gridmix unit test should validate
input/output bytes
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1061:
-------------------------------------
Status: Open (was: Patch Available)
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch, M1061-1.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1061) Gridmix unit test should validate
input/output bytes
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1061:
-------------------------------------
Status: Patch Available (was: Open)
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1061) Gridmix unit test should validate
input/output bytes
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1061:
-------------------------------------
Status: Open (was: Patch Available)
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch, M1061-1.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1061) Gridmix unit test should
validate input/output bytes
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764310#action_12764310 ]
Hong Tang commented on MAPREDUCE-1061:
--------------------------------------
Sure. As long as it is not an obvious oversight, I am fine with tightening up the test in a separate issue.
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch, M1061-1.patch, M1061-2.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1061) Gridmix unit test should
validate input/output bytes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762551#action_12762551 ]
Hadoop QA commented on MAPREDUCE-1061:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12421385/1061-0.patch
against trunk revision 819740.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/6/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/6/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/6/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/6/console
This message is automatically generated.
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1061) Gridmix unit test should
validate input/output bytes
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764309#action_12764309 ]
Chris Douglas commented on MAPREDUCE-1061:
------------------------------------------
Thanks for the review
bq. why do we need to set the actual extra bytes and records proportional to nMaps and nReds
If the spec expects 0 bytes/records, then the necessary spec data for each reduce needs to be forgiven. The amount of extra data will be proportional to the number of maps/reduces.
However, this is adjacent to some sloppiness in the map output, where the spec data is not written as part of the output, but rather as overhead. While the special case will still exist, right now it's the case for all jobs. Since the test still needs to tolerate the 0 cases, I was planning to tighten up the shuffle in a separate issue.
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch, M1061-1.patch, M1061-2.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1061) Gridmix unit test should validate
input/output bytes
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1061:
-------------------------------------
Attachment: M1061-2.patch
+/\- 1 output record for maps accounted for in tolerance
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch, M1061-1.patch, M1061-2.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1061) Gridmix unit test should validate
input/output bytes
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1061:
-------------------------------------
Status: Patch Available (was: Open)
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch, M1061-1.patch, M1061-2.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1061) Gridmix unit test should
validate input/output bytes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767044#action_12767044 ]
Hudson commented on MAPREDUCE-1061:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #80 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/80/])
. Add unit test validating byte specifications for gridmix jobs.
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch, M1061-1.patch, M1061-2.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1061) Gridmix unit test should validate
input/output bytes
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1061:
-------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I committed this.
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch, M1061-1.patch, M1061-2.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1061) Gridmix unit test should validate
input/output bytes
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-1061:
-------------------------------------
Attachment: M1061-1.patch
Updated based on feedback from Hong. Changed the checks to be per-task rather than looking at the whole job. Tolerances are proportional to the number of map/reduce tasks for bytes, 1 record for input/output.
Also tightened the reduce output bytes and improved reporting for errors during startup.
> Gridmix unit test should validate input/output bytes
> ----------------------------------------------------
>
> Key: MAPREDUCE-1061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Affects Versions: 0.21.0
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.21.0
>
> Attachments: 1061-0.patch, M1061-1.patch
>
>
> TestGridmixSubmission currently verifies only that the correct number of jobs have been run. The test should validate the I/O parameters it claims to satisfy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.