You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Sharad Agarwal (JIRA)" <ji...@apache.org> on 2008/07/25 13:07:31 UTC
[jira] Created: (HADOOP-3828) Write skipped records' bytes to DFS
Write skipped records' bytes to DFS
-----------------------------------
Key: HADOOP-3828
URL: https://issues.apache.org/jira/browse/HADOOP-3828
Project: Hadoop Core
Issue Type: New Feature
Components: mapred
Reporter: Sharad Agarwal
Assignee: Sharad Agarwal
This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-3828:
--------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Sharad!
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch, 3828_v2.patch, 3828_v3.patch, 3828_v4.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3828:
-----------------------------------
Fix Version/s: 0.19.0
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3828:
-----------------------------------
Status: Patch Available (was: Open)
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch, 3828_v2.patch, 3828_v3.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3828:
-----------------------------------
Status: Patch Available (was: Open)
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch, 3828_v2.patch, 3828_v3.patch, 3828_v4.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12626550#action_12626550 ]
Devaraj Das commented on HADOOP-3828:
-------------------------------------
Minor comments:
1) The SortedRanges class should be made package private (though this is not directly related to the patch)
2) It may make sense to enable compression for the records written to the dfs
3) In some places the indentation needs to be fixed.
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch, 3828_v2.patch, 3828_v3.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3828:
-----------------------------------
Release Note: Skipped records can optionally be written to the HDFS. Refer org.apache.hadoop.mapred.SkipBadRecords.setSkipOutputPath for setting the output path.
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch, 3828_v2.patch, 3828_v3.patch, 3828_v4.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3828:
-----------------------------------
Status: Patch Available (was: Open)
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12626523#action_12626523 ]
Sharad Agarwal commented on HADOOP-3828:
----------------------------------------
Test failure not due to this patch. TestMiniMRDFSSort.testMapReduceSort is failing due to HADOOP-3950.
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch, 3828_v2.patch, 3828_v3.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3828:
-----------------------------------
Status: Open (was: Patch Available)
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch, 3828_v2.patch, 3828_v3.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3828:
-----------------------------------
Attachment: 3828_v2.patch
Fixed the counter REDUCE_SKIPPED_RECORDS and added REDUCE_SKIPPED_GROUPS.
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch, 3828_v2.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627486#action_12627486 ]
Hudson commented on HADOOP-3828:
--------------------------------
Integrated in Hadoop-trunk #589 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/589/])
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch, 3828_v2.patch, 3828_v3.patch, 3828_v4.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12626723#action_12626723 ]
Hadoop QA commented on HADOOP-3828:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12389090/3828_v4.patch
against trunk revision 689913.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3138/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3138/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3138/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3138/console
This message is automatically generated.
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch, 3828_v2.patch, 3828_v3.patch, 3828_v4.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3828:
-----------------------------------
Status: Open (was: Patch Available)
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch, 3828_v2.patch, 3828_v3.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3828:
-----------------------------------
Attachment: 3828_v1.patch
This works as follows:-
Write the skipped record (key,value) as SequenceFile.
By default the skipped records are written in the folder "_skip" in the output dir. This is configurable using SkipBadRecords.setSkipOutputPath
-The patch also fixes a corner case by initializing the variable "skipping" in TaskInProgress.
-Also it makes some changes in SortedRanges. Made it cloneable and fixed serialization of member variable.
-cleanup in MapTask by having a different implementation of RecordReader for normal mode (skipping=false)
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Attachments: 3828_v1.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12625963#action_12625963 ]
Hadoop QA commented on HADOOP-3828:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12388896/3828_v3.patch
against trunk revision 689333.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3114/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3114/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3114/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3114/console
This message is automatically generated.
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch, 3828_v2.patch, 3828_v3.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3828:
-----------------------------------
Attachment: 3828_v4.patch
Incorporated Devaraj's comments.
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch, 3828_v2.patch, 3828_v3.patch, 3828_v4.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3828:
-----------------------------------
Attachment: 3828_v3.patch
- fixed findbugs warnings.
- default location of skip directory is now {outpurDir}/_logs/skip
- Incorporated offline comments by Devaraj to remove changes unrelated to this patch.
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch, 3828_v2.patch, 3828_v3.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3828) Write skipped records' bytes to DFS
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624850#action_12624850 ]
Hadoop QA commented on HADOOP-3828:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12388592/3828_v2.patch
against trunk revision 687868.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 4 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3083/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3083/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3083/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3083/console
This message is automatically generated.
> Write skipped records' bytes to DFS
> -----------------------------------
>
> Key: HADOOP-3828
> URL: https://issues.apache.org/jira/browse/HADOOP-3828
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3828_v1.patch, 3828_v2.patch
>
>
> This is an incremental step over HADOOP-153, which provides the base skipping functionality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.