You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/05/04 19:35:36 UTC
[GitHub] [spark] MaxGekk commented on a diff in pull request #36435: [SPARK-35912][SQL][FOLLOW-UP] Add a legacy configuration for respecting nullability in DataFrame.schema.csv/json(ds)
MaxGekk commented on code in PR #36435:
URL: https://github.com/apache/spark/pull/36435#discussion_r865208393
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3025,6 +3025,18 @@ object SQLConf {
.intConf
.createOptional
+ val LEGACY_RESPECT_NULLABILITY_IN_TEXT_DATASET_CONVERSION =
+ buildConf("spark.sql.legacy.respectNullabilityInTextDatasetConversion")
Review Comment:
Looking configs around, this should be shifted by 2 spaces left?
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala:
##########
@@ -2693,6 +2693,17 @@ abstract class CSVSuite
assert(df.schema == expected)
checkAnswer(df, Row(1, null) :: Nil)
}
+
+ withSQLConf(
+ SQLConf.LEGACY_RESPECT_NULLABILITY_IN_TEXT_DATASET_CONVERSION.key -> true.toString) {
Review Comment:
nit:
```suggestion
withSQLConf(SQLConf.LEGACY_RESPECT_NULLABILITY_IN_TEXT_DATASET_CONVERSION.key -> "true") {
```
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3025,6 +3025,18 @@ object SQLConf {
.intConf
.createOptional
+ val LEGACY_RESPECT_NULLABILITY_IN_TEXT_DATASET_CONVERSION =
+ buildConf("spark.sql.legacy.respectNullabilityInTextDatasetConversion")
+ .internal()
+ .doc(
+ "When true, the nullability in the user-specified schema for " +
+ "`DataFrameReader.schema(schema).json(jsonDataset)` and " +
+ "`DataFrameReader.schema(schema).csv(csvDataset)` is respected. Otherwise, they are " +
+ "turned to a nullable schema forcely.")
Review Comment:
unindent by 2 spaces
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3025,6 +3025,18 @@ object SQLConf {
.intConf
.createOptional
+ val LEGACY_RESPECT_NULLABILITY_IN_TEXT_DATASET_CONVERSION =
+ buildConf("spark.sql.legacy.respectNullabilityInTextDatasetConversion")
+ .internal()
+ .doc(
+ "When true, the nullability in the user-specified schema for " +
+ "`DataFrameReader.schema(schema).json(jsonDataset)` and " +
+ "`DataFrameReader.schema(schema).csv(csvDataset)` is respected. Otherwise, they are " +
+ "turned to a nullable schema forcely.")
Review Comment:
forcely -> forcibly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org