You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Ashutosh Gupta (Jira)" <ji...@apache.org> on 2022/06/30 15:42:00 UTC
[jira] [Created] (HADOOP-18321) Fix when to read an additional record from a BZip2 text file split
Ashutosh Gupta created HADOOP-18321:
---------------------------------------
Summary: Fix when to read an additional record from a BZip2 text file split
Key: HADOOP-18321
URL: https://issues.apache.org/jira/browse/HADOOP-18321
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.3.3
Reporter: Ashutosh Gupta
Assignee: Ashutosh Gupta
Fix data correctness issue with TextInputFormat that can occur when reading BZip2 compressed text files. When triggered this bug would cause a split to return the first record of the succeeding split that reads the next BZip2 block, thereby duplicating that record.
*When the bug is triggered?*
The condition for the bug to occur requires that the flag "needAdditionalRecord" in CompressedSplitLineReader to be set to true by #fillBuffer at an inappropriate time: when we haven't read the remaining bytes of split. This can happen when the inDelimiter parameter is true while #fillBuffer is invoked while reading the next line. The inDelimiter parameter is true when either 1) the last byte of the buffer is a CR character ('\r') if using the default delimiters, or 2) the last bytes of the buffer are a common prefix of the delimiter if using a custom delimiter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org