[GitHub] [spark] wayneguow commented on pull request #34860: [SPARK-37604][SQL] Change the behavior of emptyValueInRead parameter in CSVOptions

[GitBox <gi...@apache.org>]

wayneguow commented on pull request #34860:
URL: https://github.com/apache/spark/pull/34860#issuecomment-991860440


   Yes, I agree with you that it would make a breaking change. The parameter `emptyValue` which designed in `CsvParserSettings` of Univocity is used instead of an empty strings when reading csv files.
   
   What makes me confused is that is there any better way to read csv files which are written with self-defined emptyValue. In writing, if emptyValue is set to "EMPTY", when reading, we can't recognize empty strings("EMPTY") but got a "EMPTY" string rather "". We need to handle "EMPTY" strings with hardcode rather than setting options when reading.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org