You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-issues@hadoop.apache.org by "Junping Du (JIRA)" <ji...@apache.org> on 2015/12/01 13:02:11 UTC

[jira] [Updated] (MAPREDUCE-6558) multibyte delimiters with compressed input files generate duplicate records

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Junping Du updated MAPREDUCE-6558:
----------------------------------
    Target Version/s: 2.8.0, 2.7.3, 2.6.4  (was: 2.8.0, 2.6.3, 2.7.3)

> multibyte delimiters with compressed input files generate duplicate records
> ---------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6558
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6558
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv1, mrv2
>    Affects Versions: 2.7.2
>            Reporter: Wilfred Spiegelenburg
>            Assignee: Wilfred Spiegelenburg
>
> This is the follow up for MAPREDUCE-6549. Compressed files cause record duplications as shown in different junit tests. The number of duplicated records changes with the splitsize:
> Unexpected number of records in split (splitsize = 10)
> Expected: 41051
> Actual: 45062
> Unexpected number of records in split (splitsize = 100000)
> Expected: 41051
> Actual: 41052
> Test passes with splitsize = 147445 which is the compressed file length.The file is a bzip2 file with 100k blocks and a total of 11 blocks



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)