[jira] [Created] (HADOOP-18321) Fix when to read an additional record from a BZip2 text file split

["Ashutosh Gupta (Jira)" <ji...@apache.org>]

Ashutosh Gupta created HADOOP-18321:
---------------------------------------

             Summary: Fix when to read an additional record from a BZip2 text file split
                 Key: HADOOP-18321
                 URL: https://issues.apache.org/jira/browse/HADOOP-18321
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 3.3.3
            Reporter: Ashutosh Gupta
            Assignee: Ashutosh Gupta


Fix data correctness issue with TextInputFormat that can occur when reading BZip2 compressed text files. When triggered this bug would cause a split to return the first record of the succeeding split that reads the next BZip2 block, thereby duplicating that record.

*When the bug is triggered?*

The condition for the bug to occur requires that the flag "needAdditionalRecord" in CompressedSplitLineReader to be set to true by #fillBuffer at an inappropriate time: when we haven't read the remaining bytes of split. This can happen when the inDelimiter parameter is true while #fillBuffer is invoked while reading the next line. The inDelimiter parameter is true when either 1) the last byte of the buffer is a CR character ('\r') if using the default delimiters, or 2) the last bytes of the buffer are a common prefix of the delimiter if using a custom delimiter.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org