[jira] [Work logged] (HADOOP-18321) Fix when to read an additional record from a BZip2 text file split

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-18321?focusedWorklogId=786695&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-786695 ]

ASF GitHub Bot logged work on HADOOP-18321:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 30/Jun/22 16:18
            Start Date: 30/Jun/22 16:18
    Worklog Time Spent: 10m 
      Work Description: ashutoshcipher opened a new pull request, #4521:
URL: https://github.com/apache/hadoop/pull/4521

   ### Description of PR
   
   Fix when to read an additional record from a BZip2 text file split
   
   JIRA - HADOOP-18321
   
   ### How was this patch tested?
   
   Added Units
   
   
   ### For code changes:
   
   - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
   
   




Issue Time Tracking
-------------------

            Worklog Id:     (was: 786695)
    Remaining Estimate: 0h
            Time Spent: 10m

> Fix when to read an additional record from a BZip2 text file split
> ------------------------------------------------------------------
>
>                 Key: HADOOP-18321
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18321
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 3.3.3
>            Reporter: Ashutosh Gupta
>            Assignee: Ashutosh Gupta
>            Priority: Critical
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Fix data correctness issue with TextInputFormat that can occur when reading BZip2 compressed text files. When triggered this bug would cause a split to return the first record of the succeeding split that reads the next BZip2 block, thereby duplicating that record.
> *When the bug is triggered?*
> The condition for the bug to occur requires that the flag "needAdditionalRecord" in CompressedSplitLineReader to be set to true by #fillBuffer at an inappropriate time: when we haven't read the remaining bytes of split. This can happen when the inDelimiter parameter is true while #fillBuffer is invoked while reading the next line. The inDelimiter parameter is true when either 1) the last byte of the buffer is a CR character ('\r') if using the default delimiters, or 2) the last bytes of the buffer are a common prefix of the delimiter if using a custom delimiter.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org