You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Jason Lowe (JIRA)" <ji...@apache.org> on 2013/06/05 23:35:20 UTC
[jira] [Comment Edited] (HADOOP-9622) bzip2 codec can drop records
when reading data in splits
[ https://issues.apache.org/jira/browse/HADOOP-9622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13676373#comment-13676373 ]
Jason Lowe edited comment on HADOOP-9622 at 6/5/13 9:33 PM:
------------------------------------------------------------
Attaching a testcase for the problem, which requires the same input file as used in one of Pig's bzip2 unit tests, [blockEndingInCR.txt.bz2|https://issues.apache.org/jira/secure/attachment/12438881/blockEndingInCR.txt.bz2]. Drop the file into hadoop-common-project/hadoop-common/src/test/resources/ and the unit test can be run via Maven.
I've also manually verified the loss of records via a simple streaming job:
{noformat}
hadoop jar hadoop-*/share/hadoop/tools/lib/hadoop-streaming*.jar -Dmapreduce.input.fileinputformat.split.minsize=136498 -Dmapreduce.job.maps=3 -input blockEndingInCR.txt.bz2 -output streamout -mapper cat -reducer NONE
{noformat}
Examining the output of the job shows that the record after the carriage return in the file was dropped. Running a similar streaming job with only one map (no splits) shows the record is not dropped.
was (Author: jlowe):
Attaching a testcase for the problem, which requires the same input file as used in one of Pig's bzip2 unit tests, [blockEndingInCR.txt.bz2|https://issues.apache.org/jira/secure/attachment/12438881/blockEndingInCR.txt.bz2]. Drop the file into hadoop-common-project/hadoop-common/src/test/resources/ and the unit test can be run via Maven.
I've also manually verified the loss of records via a simple streaming job:
hadoop jar hadoop-*/share/hadoop/tools/lib/hadoop-streaming*.jar -Dmapreduce.input.fileinputformat.split.minsize=136498 -Dmapreduce.job.maps=3 -input blockEndingInCR.txt.bz2 -output streamout -mapper cat -reducer NONE
Examining the output of the job shows that the record after the carriage return in the file was dropped. Running a similar streaming job with only one map (no splits) shows the record is not dropped.
> bzip2 codec can drop records when reading data in splits
> --------------------------------------------------------
>
> Key: HADOOP-9622
> URL: https://issues.apache.org/jira/browse/HADOOP-9622
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 2.0.4-alpha, 0.23.8
> Reporter: Jason Lowe
> Priority: Critical
> Attachments: HADOOP-9622-testcase.patch
>
>
> Bzip2Codec.BZip2CompressionInputStream can cause records to be dropped when reading them in splits based on where record delimiters occur relative to compression block boundaries.
> Thanks to [~knoguchi] for discovering this problem while working on PIG-3251.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira