You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/16 04:17:41 UTC
[GitHub] [spark] wayneguow commented on pull request #34860: [SPARK-37604][SQL] Change emptyValueInRead's effect to that any fields matching this string will be set as "" when reading csv files
wayneguow commented on pull request #34860:
URL: https://github.com/apache/spark/pull/34860#issuecomment-995425112
Maybe a kind of possible approach to avoid a breaking change is that adding another boolean option in CSVOptions such as `parseEmptyValueAsEmpty` which is default set as false. And with this option, we can change the `nullSafeDatum` method in UnivocityParser as follows:
```
private def nullSafeDatum(
datum: String,
name: String,
nullable: Boolean,
options: CSVOptions)(converter: ValueConverter): Any = {
if (datum == options.nullValue || datum == null) {
if (!nullable) {
throw new RuntimeException(s"null value found but field $name is not nullable.")
}
null
} else if (options.parseEmptyValueAsEmpty && datum == options.emptyValueInRead) {
converter.apply("")
} else {
converter.apply(datum)
}
}
```
With this option, the default behavior is same with before, but for users such as me, we can succeed to parse emptyValue strings to "" when reading csv files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org