[GitHub] [carbondata] jack86596 opened a new pull request #3664: [CARBONDATA-3740] Add line separator option to load command to configure the line separator during csv parsing.

[GitBox <gi...@apache.org>]

jack86596 opened a new pull request #3664: [CARBONDATA-3740] Add line separator option to load command to configure the line separator during csv parsing.
URL: https://github.com/apache/carbondata/pull/3664
 
 
    ### Why is this PR needed?
    Sometime univocity parser will detect the line separator incorrectly. In this case, user should be able to set line separator explicitly.
    Issue: During loading, if in the first line, there is one field has a '\r' character and this '\r' appears before the first '\n', line separator detection will treat '\r' as line separator. This is not the intention.
    Example:
    Data file has two line, ^M is '\r':
    1,2^M,3
    4,5,6
    After loading,
    The records in table will be:
    | 1 | 2 | null |
    | null | 3
    4 | 5 |
    Correct should be:
     | 1 | 2^M | 3 |
     | 4 | 5 | 6 |
    
    ### What changes were proposed in this PR?
    Allow user to specify line separator explicitly in load command, add one new option to load command named "line_separator".
       
    ### Does this PR introduce any user interface change?
     - Yes. New load option "line_separator" is added. 
   
    ### Is any new testcase added?
     - Yes. 
   
       
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services