[GitHub] spark issue #20877: [SPARK-23765][SQL] Supports custom line separator for js...

[HyukjinKwon <gi...@git.apache.org>]

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20877
  
    @MaxGekk, are you busy? Do you have some time to go for CSV's lineSep? I think I wouldn't have some time within a couple of weeks. If you have some time, I would appreciate if you could go ahead. Otherwise, I will try this one after a couple of weeks.
    
    The problem in CSV's lineSep is about multiline support. As you might already know, CSV's multiline mode is different with JSON in a way it parses line by line from the stream whereas JSON treats it as a whole record in general - so we should set the lineSep to Univocity parser as well.
    
    The problem is, `lineSep` at Univocity parser has some limitation (https://github.com/apache/spark/pull/18581#issuecomment-314037750 and see also `https://github.com/uniVocity/univocity-parsers/issues/170`).
    
    There are some changes made in https://github.com/apache/spark/pull/18581 . Might able to extract CSV related change and make some addition and deletion.
    
    If it's difficult to support `lineSep` more than one characters by the limitation, I think we can restrict the lineSep only to one character in `multiLine` mode.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org