You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/16 03:41:00 UTC

[GitHub] [spark] wayneguow edited a comment on pull request #34860: [SPARK-37604][SQL] Change emptyValueInRead's effect to that any fields matching this string will be set as "" when reading csv files

wayneguow edited a comment on pull request #34860:
URL: https://github.com/apache/spark/pull/34860#issuecomment-991860440


   Yes, I agree with you that it would make a breaking change. The parameter `emptyValue` which designed in `CsvParserSettings` of Univocity is used instead of an empty strings when reading csv files.
   
   What makes me confused is that is there any better way to read csv files which are written with self-defined emptyValue. In writing, if emptyValue is set to "EMPTY", when reading, we can't recognize empty strings("EMPTY") but got a "EMPTY" string rather than "". We need to handle "EMPTY" strings with hardcode rather than setting options when reading.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org