[jira] [Created] (MAPREDUCE-6598) LineReader enhencement to support text records contains "\n"

["cloudyarea (JIRA)" <ji...@apache.org>]

cloudyarea created MAPREDUCE-6598:
-------------------------------------

             Summary: LineReader enhencement to support text records contains "\n"
                 Key: MAPREDUCE-6598
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6598
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: mrv2
    Affects Versions: 2.6.0
         Environment: RHEL 7, Spark 1.3.1, Hadoop 2.6.0
            Reporter: cloudyarea
            Priority: Minor


We have billions of XML message records stored on text files need to be parsed parallel by Spark. By default, Spark open a Hadoop text file using LineReader which provides a single line of text as a record. 

The XML messages contains "\n" and I believe it is a common scenario - many users have cross-line records. Currently, the solution is to the extend the interface RecordReader.

To reduce the repeat work, I wrote a class named MessageRecordReader to extend the interface RecordReader, user can set a string as record delimiter, then MessageRecordReader provides a multiple line record to user. 

I would like to contribute the code to community. Please let me know if you are interested in this simple but useful implementation. 

Thank you very much and happy new year!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)