You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-issues@hadoop.apache.org by "zhihai xu (JIRA)" <ji...@apache.org> on 2014/05/23 10:54:02 UTC

[jira] [Commented] (MAPREDUCE-5777) Support utf-8 text with BOM (byte order marker)

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14006984#comment-14006984 ] 

zhihai xu commented on MAPREDUCE-5777:
--------------------------------------

I just upload a patch for review. This patch included a test case to verify BOM skipped by LineRecordRead.
This patch also add a utf-8 text file with BOM which is used by the new test case:testStripBOM.
The new test case is passed with this change in LineRecordRead and the new case case will fail with original code.

> Support utf-8 text with BOM (byte order marker)
> -----------------------------------------------
>
>                 Key: MAPREDUCE-5777
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5777
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.22.0, 2.2.0
>            Reporter: bc Wong
>            Assignee: zhihai xu
>         Attachments: mapreduce-5777.patch
>
>
> UTF-8 text may have a BOM. TextInputFormat, KeyValueTextInputFormat and friends should recognize the BOM and not treat it as actual data.



--
This message was sent by Atlassian JIRA
(v6.2#6252)