You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Chris Douglas (JIRA)" <ji...@apache.org> on 2009/04/11 02:18:14 UTC
[jira] Created: (HADOOP-5657) Validate data passed through
TestReduceFetch
Validate data passed through TestReduceFetch
--------------------------------------------
Key: HADOOP-5657
URL: https://issues.apache.org/jira/browse/HADOOP-5657
Project: Hadoop Core
Issue Type: Improvement
Components: mapred, test
Reporter: Chris Douglas
While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5657:
----------------------------------
Status: Open (was: Patch Available)
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Minor
> Attachments: 5657-0.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700022#action_12700022 ]
Devaraj Das commented on HADOOP-5657:
-------------------------------------
+1
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Minor
> Attachments: 5657-0.patch, 5657-1.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5657:
----------------------------------
Assignee: Chris Douglas
Status: Patch Available (was: Open)
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Attachments: 5657-0.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700051#action_12700051 ]
Hadoop QA commented on HADOOP-5657:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12405486/5657-1.patch
against trunk revision 765815.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/console
This message is automatically generated.
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Minor
> Attachments: 5657-0.patch, 5657-1.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5657:
----------------------------------
Attachment: (was: 5657-0.patch)
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Minor
> Attachments: 5657-0.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704283#action_12704283 ]
Hudson commented on HADOOP-5657:
--------------------------------
Integrated in Hadoop-trunk #821 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/821/])
. Validate data in TestReduceFetch to improve merge test coverage.
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: 5657-0.patch, 5657-1.patch, 5657-2.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5657:
----------------------------------
Attachment: (was: 5657-2.patch)
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Minor
> Attachments: 5657-0.patch, 5657-1.patch, 5657-2.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5657:
----------------------------------
Attachment: 5657-0.patch
The following are now validated in the reduce:
* Each map produces one record for each of 4096 small keys
* Includes unique large records, each straddled by a pair of small records from another map (to detect corruption from the merge)
* Changes some parameters for {{testReduceFromDisk}} to make intermediate merges with in-memory data occur occasionally
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Attachments: 5657-0.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700023#action_12700023 ]
Devaraj Das commented on HADOOP-5657:
-------------------------------------
It'd help code readability if some comments are added on why two values are emitted per map and the logic the testcase is employing for validation (verbose comments for the arithmetic is what I mean *smile*).
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Minor
> Attachments: 5657-0.patch, 5657-1.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5657:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I committed this.
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: 5657-0.patch, 5657-1.patch, 5657-2.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5657:
----------------------------------
Attachment: 5657-1.patch
Emits only two additional records per map; less expensive key updates. Detects known-bad case when merging a combination of in-memory and on-disk segments.
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Minor
> Attachments: 5657-0.patch, 5657-1.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5657:
----------------------------------
Attachment: 5657-0.patch
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Minor
> Attachments: 5657-0.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5657:
----------------------------------
Attachment: 5657-2.patch
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Minor
> Attachments: 5657-0.patch, 5657-1.patch, 5657-2.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5657:
----------------------------------
Status: Patch Available (was: Open)
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Minor
> Attachments: 5657-0.patch, 5657-1.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12698296#action_12698296 ]
Hadoop QA commented on HADOOP-5657:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12405214/5657-0.patch
against trunk revision 764287.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/console
This message is automatically generated.
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Minor
> Attachments: 5657-0.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5657:
----------------------------------
Attachment: 5657-2.patch
Adds some extra comments; no functional changes
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Minor
> Attachments: 5657-0.patch, 5657-1.patch, 5657-2.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5657) Validate data passed through
TestReduceFetch
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5657:
----------------------------------
Priority: Minor (was: Major)
> Validate data passed through TestReduceFetch
> --------------------------------------------
>
> Key: HADOOP-5657
> URL: https://issues.apache.org/jira/browse/HADOOP-5657
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, test
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Priority: Minor
> Attachments: 5657-0.patch
>
>
> While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.