You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Sharad Agarwal (JIRA)" <ji...@apache.org> on 2008/07/25 13:17:31 UTC
[jira] Created: (HADOOP-3829) Narrown down skipped records based on
user acceptable value
Narrown down skipped records based on user acceptable value
-----------------------------------------------------------
Key: HADOOP-3829
URL: https://issues.apache.org/jira/browse/HADOOP-3829
Project: Hadoop Core
Issue Type: New Feature
Components: mapred
Reporter: Sharad Agarwal
Assignee: Sharad Agarwal
This is an incremental step over HADOOP-153.
If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3829) Narrown down skipped records based on
user acceptable value
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3829:
-----------------------------------
Status: Open (was: Patch Available)
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch, 3829_v3.patch, 3829_v4.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3829) Narrown down skipped records based
on user acceptable value
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631755#action_12631755 ]
Sharad Agarwal commented on HADOOP-3829:
----------------------------------------
ant test passed on my machine.
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch, 3829_v3.patch, 3829_v4.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3829) Narrown down skipped records based
on user acceptable value
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632079#action_12632079 ]
Hadoop QA commented on HADOOP-3829:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12390255/3829_v4.patch
against trunk revision 696525.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 11 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3293/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3293/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3293/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3293/console
This message is automatically generated.
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch, 3829_v3.patch, 3829_v4.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3829) Narrown down skipped records based on
user acceptable value
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3829:
-----------------------------------
Attachment: 3829_v4.patch
Fixed an issue in ReduceTask#SkippingReduceValuesIterator.
Added more documentation to SkipBadRecords.
Made writing of skip records optional.
Incorporated Devaraj's offline comment to remove the SkipBadRecords.ENABLED flag as it is now redundant after the addition of MAPPER_MAX_SKIP_RECORDS/REDUCER_MAX_SKIP_GROUPS.
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch, 3829_v3.patch, 3829_v4.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3829) Narrown down skipped records based on
user acceptable value
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3829:
-----------------------------------
Attachment: 3829_v2.patch
fairly tested patch. few additional things:
- moved counters from Counters.java to SkipBadRecords.java as these are specific to skip feature.
- fixed hasNext in SortedRanges.SkipRangeIterator.
- skipped records are not written to HDFS, if it is a test attempt (test attempt is to figure out whether a range is good or bad. In this only records in test range are passed to the mapper/reducer, others are skipped).
- recordreader.next not called beyond the test range during test attempt.
- renamed failedRanges to skipRanges in Task.java
- added config params to hadoop-default
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3829) Narrown down skipped records based on
user acceptable value
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3829:
-----------------------------------
Release Note: If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value. Use org.apache.hadoop.mapred.SkipBadRecords.setMapperMaxSkipRecords for setting the acceptable value.
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch, 3829_v3.patch, 3829_v4.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3829) Narrown down skipped records based on
user acceptable value
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3829:
-----------------------------------
Status: Patch Available (was: Open)
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch, 3829_v3.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3829) Narrown down skipped records based on
user acceptable value
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3829:
-----------------------------------
Attachment: 3829_v3.patch
updated with the trunk.
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch, 3829_v3.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3829) Narrown down skipped records based
on user acceptable value
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633311#action_12633311 ]
Hudson commented on HADOOP-3829:
--------------------------------
Integrated in Hadoop-trunk #611 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/611/])
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch, 3829_v3.patch, 3829_v4.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3829) Narrown down skipped records based on
user acceptable value
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3829:
-----------------------------------
Status: Patch Available (was: Open)
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch, 3829_v3.patch, 3829_v4.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3829) Narrown down skipped records based on
user acceptable value
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-3829:
--------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Sharad!
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch, 3829_v3.patch, 3829_v4.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3829) Narrown down skipped records based on
user acceptable value
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3829:
-----------------------------------
Affects Version/s: 0.19.0
Fix Version/s: 0.19.0
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch, 3829_v3.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3829) Narrown down skipped records based
on user acceptable value
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631724#action_12631724 ]
Sharad Agarwal commented on HADOOP-3829:
----------------------------------------
test-patch passed on my machine.
[exec] +1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] +1 tests included. The patch appears to include 11 new or modified tests.
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch, 3829_v3.patch, 3829_v4.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3829) Narrown down skipped records based on
user acceptable value
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3829:
-----------------------------------
Attachment: 3829_v1.153_7.patch
This patch depends on the patch from HADOOP-153. Please apply 153_7.patch before applying this one.
The approach has been discussed earlier in HADOOP-153 as well. Here is the brief:
Defines user configurable MAPPER_MAX_SKIP_RECORDS/MAPPER_REDUCE_SKIP_RECORDS -> acceptable skipped records in the neighborhood of a bad record.
If skipped range is greater than this threshold, the task will try to narrow down the skipped range using a binary search kind of algorithm during task re-executions till this threshold is met or all task attempts are exhausted. The skipped range is divided into two halves and only one half get executed. Based on the subsequent failure, it figures out which half contains the bad record.
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Attachments: 3829_v1.153_7.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3829) Narrown down skipped records based
on user acceptable value
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629129#action_12629129 ]
Hadoop QA commented on HADOOP-3829:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12389656/3829_v3.patch
against trunk revision 692996.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3205/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3205/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3205/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3205/console
This message is automatically generated.
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch, 3829_v3.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3829) Narrown down skipped records based
on user acceptable value
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632091#action_12632091 ]
Sharad Agarwal commented on HADOOP-3829:
----------------------------------------
TestFileAppend2.testComplexAppend has failed on Hudson which is unrelated to this patch.
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch, 3829_v3.patch, 3829_v4.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3829) Narrown down skipped records based on
user acceptable value
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sharad Agarwal updated HADOOP-3829:
-----------------------------------
Attachment: 3829_v1.patch
Attaching the working patch, while I continue to test.
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3829) Narrown down skipped records based on
user acceptable value
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3829:
------------------------------------
Release Note: Introduced new config parameter org.apache.hadoop.mapred.SkipBadRecords.setMapperMaxSkipRecords to set range of records to be skipped in the neighborhood of a failed record. (was: If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value. Use org.apache.hadoop.mapred.SkipBadRecords.setMapperMaxSkipRecords for setting the acceptable value.)
> Narrown down skipped records based on user acceptable value
> -----------------------------------------------------------
>
> Key: HADOOP-3829
> URL: https://issues.apache.org/jira/browse/HADOOP-3829
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Sharad Agarwal
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3829_v1.153_7.patch, 3829_v1.patch, 3829_v2.patch, 3829_v3.patch, 3829_v4.patch
>
>
> This is an incremental step over HADOOP-153.
> If the number of skipped records in the neighborhood of a bad record are not acceptable to the user, then narrow down the skipped range to the user acceptable value.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.