You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2026/09/04 16:20:03 UTC
(spark) branch master updated: [SPARK-58917][SQL] Add option to respect `inferSchema` when reading CSV as variant
This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3a0845a58810 [SPARK-58917][SQL] Add option to respect `inferSchema` when reading CSV as variant
3a0845a58810 is described below
commit 3a0845a5881055a4e480cdb0fcdcd060d04adb95
Author: Greg Hansen <gr...@databricks.com>
AuthorDate: Sat Sep 5 00:19:36 2026 +0800
[SPARK-58917][SQL] Add option to respect `inferSchema` when reading CSV as variant
### What changes were proposed in this pull request?
This PR adds a new CSV read option, `variantRespectInferSchema` (default False). When set to True and `inferSchema` is False, the CSV-to-Variant parser preserves scalar leaf values as strings instead of inferring their types. This applies wherever CSV values are read into a VARIANT:
* When using the `singleVariantColumn` option
* When reading explicitly-typed VariantType columns
#### Decision notes
* We use a per-read reader option to allow users to set this with `.option(...)` instead of as a global Spark config.
* The default is False to preserve the existing best-effort type inference unless a user opts-in.
### Why are the changes needed?
When reading CSV records as Variant, the parser infers scalar types regardless of `inferSchema`. Text can be changed by type inference (e.g. "0001" is stored as the integer 1, "+5" as 5, "True" as boolean true). There is no way to retain the original string values.
See [JIRA issue](https://issues.apache.org/jira/browse/SPARK-58917) for more details.
### Does this PR introduce _any_ user-facing change?
Yes — a new read option variantRespectInferSchema. Default false, so no change for existing workloads. With it enabled:
```scala
spark
.read
.option("singleVariantColumn", "v")
.option("variantRespectInferSchema", "true")
.csv(path)
```
### How was this patch tested?
Added a unit test in CSVSuite which is passing on CSVv1Suite and CSVv2Suite.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
Closes #58175 from ghanse/csv-variant-scalar-as-string.
Authored-by: Greg Hansen <gr...@databricks.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
docs/sql-data-sources-csv.md | 10 +-
.../apache/spark/sql/catalyst/csv/CSVOptions.scala | 4 +
.../spark/sql/catalyst/csv/UnivocityParser.scala | 5 +
.../org/apache/spark/sql/CsvFunctionsSuite.scala | 41 +++++++
.../sql/execution/datasources/csv/CSVSuite.scala | 122 ++++++++++++++++++++-
5 files changed, 179 insertions(+), 3 deletions(-)
diff --git a/docs/sql-data-sources-csv.md b/docs/sql-data-sources-csv.md
index bf63fe2be5d3..6fe3dbd83af7 100644
--- a/docs/sql-data-sources-csv.md
+++ b/docs/sql-data-sources-csv.md
@@ -111,7 +111,7 @@ Data source options of CSV can be set via:
<tr>
<td><code>inferSchema</code></td>
<td>false</td>
- <td>Infers the input schema automatically from data. It requires one extra pass over the data. CSV built-in functions ignore this option.</td>
+ <td>Infers the input schema automatically from data. It requires one extra pass over the data. CSV built-in functions ignore this option, except as noted under <code>variantRespectInferSchema</code>.</td>
<td>read</td>
</tr>
<tr>
@@ -225,7 +225,13 @@ Data source options of CSV can be set via:
<tr>
<td><code>singleVariantColumn</code></td>
<td>(none)</td>
- <td>If specified, the entire CSV record is parsed and stored as a single column of <code>VariantType</code> with the given column name, instead of being split into individual fields.</td>
+ <td>If specified, the entire CSV record is parsed and stored as a single column of <code>VariantType</code> with the given column name, instead of being split into individual fields. By default, scalar values are type-inferred inside the variant on a best-effort basis (for example, <code>"0001"</code> may be inferred as the integer <code>1</code>). To preserve scalar values as strings, set <code>variantRespectInferSchema</code> to <code>true</code> and <code>inferSchema</code> to <co [...]
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>variantRespectInferSchema</code></td>
+ <td>false</td>
+ <td>When reading into a <code>VARIANT</code> (see <code>singleVariantColumn</code> or a <code>VariantType</code> column in the schema), controls whether the <code>inferSchema</code> option is honored. When <code>true</code> and <code>inferSchema</code> is <code>false</code>, scalar values are preserved as strings inside the variant instead of being inferred as boolean, long, decimal, date, or timestamp. When <code>false</code> (the default), scalar types are inferred on a best-effort [...]
<td>read</td>
</tr>
<tr>
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
index 7db03a8a2323..a75a2da32f13 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
@@ -337,6 +337,9 @@ class CSVOptions(
// E.g. spark.read.format("csv").option("singleVariantColumn", "colName")
val singleVariantColumn: Option[String] = parameters.get(SINGLE_VARIANT_COLUMN)
+ // When true, reading CSV values into a VARIANT honors `inferSchema` (default: false).
+ val variantRespectInferSchema = getBool(VARIANT_RESPECT_INFER_SCHEMA, default = false)
+
def needHeaderForSingleVariantColumn: Boolean =
singleVariantColumn.isDefined && headerFlag
@@ -443,6 +446,7 @@ object CSVOptions extends DataSourceOptions {
newOption(SEP, DELIMITER)
val COLUMN_PRUNING = newOption("columnPruning")
val SINGLE_VARIANT_COLUMN = newOption(DataSourceOptions.SINGLE_VARIANT_COLUMN)
+ val VARIANT_RESPECT_INFER_SCHEMA = newOption("variantRespectInferSchema")
// Max error content length in CSV parser/writer exception messages, and the bound on the bad
// record embedded in MALFORMED_CSV_RECORD errors.
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
index fa6bd19064f0..4583c328239e 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
@@ -481,6 +481,11 @@ class UnivocityParser(
return
}
+ if (!options.inferSchemaFlag && options.variantRespectInferSchema) {
+ builder.appendString(s)
+ return
+ }
+
def parseLong(): DataType = {
try {
builder.appendLong(s.toLong)
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
index 455434b4de01..f79a7535cc14 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
@@ -869,6 +869,47 @@ class CsvFunctionsSuite extends SharedSparkSession {
Seq(Row(s"""{null, $largeInput}""")))
}
+ test("from_csv with variant: variantRespectInferSchema controls scalar inference") {
+ // from_csv shares the CSV parser, so it observes inferSchema only when
+ // variantRespectInferSchema is enabled: with inferSchema off the variant scalar stays a
+ // string, otherwise it is inferred. Without the flag, from_csv ignores inferSchema entirely.
+ val df = Seq("100").toDF("value")
+
+ // With variantRespectInferSchema enabled and inferSchema off the scalar is preserved as a
+ // string. inferSchema defaults to false, so setting the flag alone behaves the same.
+ for (options <- Seq(
+ Map("singleVariantColumn" -> "v", "variantRespectInferSchema" -> "true",
+ "inferSchema" -> "false"),
+ Map("singleVariantColumn" -> "v", "variantRespectInferSchema" -> "true"))) {
+ checkAnswer(
+ df.select(from_csv($"value", StructType.fromDDL("v variant"), options).cast("string")),
+ Seq(Row("""{{"_c0":"100"}}""")))
+ }
+
+ // Without variantRespectInferSchema, the scalar is inferred regardless of inferSchema.
+ // Both the default and explicitly configured inferSchema setting should result in inferred
+ // scalar values. Values should also be inferred when both options are enabled.
+ for (options <- Seq(
+ Map("singleVariantColumn" -> "v"),
+ Map("singleVariantColumn" -> "v", "inferSchema" -> "true"),
+ Map("singleVariantColumn" -> "v", "variantRespectInferSchema" -> "true",
+ "inferSchema" -> "true"))) {
+ checkAnswer(
+ df.select(from_csv($"value", StructType.fromDDL("v variant"), options).cast("string")),
+ Seq(Row("""{{"_c0":100}}""")))
+ }
+
+ // Explicit VariantType columns should behave the same. schema_of_variant tells the string
+ // "100" from the inferred long.
+ checkAnswer(
+ df.selectExpr("schema_of_variant(from_csv(value, 'a variant', " +
+ "map('variantRespectInferSchema', 'true', 'inferSchema', 'false')).a)"),
+ Seq(Row("STRING")))
+ checkAnswer(
+ df.selectExpr("schema_of_variant(from_csv(value, 'a variant').a)"),
+ Seq(Row("BIGINT")))
+ }
+
test("from_csv with variant: extreme negative scale decimal does not hang") {
// A value like "1E99999" parses to a BigDecimal with scale=-99999.
// Calling setScale(0) on it would hang, so it should fall through to string.
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
index 3aefe25f6a08..0d0249fbad54 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
@@ -3342,7 +3342,7 @@ abstract class CSVSuite
}
test("validate CSV Options") {
- assert(CSVOptions.getAllOptions.size == 42)
+ assert(CSVOptions.getAllOptions.size == 43)
// Please add validation on any new CSV options here
assert(CSVOptions.isValidOption("header"))
assert(CSVOptions.isValidOption("inferSchema"))
@@ -3386,6 +3386,7 @@ abstract class CSVSuite
assert(CSVOptions.isValidOption("delimiter"))
assert(CSVOptions.isValidOption("singleVariantColumn"))
assert(CSVOptions.isValidOption("columnPruning"))
+ assert(CSVOptions.isValidOption("variantRespectInferSchema"))
// Please add validation on any new CSV options with alternative here
assert(CSVOptions.getAlternativeOption("sep").contains("delimiter"))
assert(CSVOptions.getAlternativeOption("delimiter").contains("sep"))
@@ -3898,6 +3899,125 @@ abstract class CSVSuite
}
}
+ test("csv variant retains scalars as strings when inferSchema is disabled") {
+ withTempPath { path =>
+ val data =
+ """field 1,field2
+ |100,1.1
+ |2000-01-01,2000-01-01 01:02:03
+ |,true
+ |1e9,hello,extra
+ |missing
+ |""".stripMargin
+ Files.write(path.toPath, data.getBytes(StandardCharsets.UTF_8))
+
+ def checkSingleVariant(options: Map[String, String], expected: String*): Unit = {
+ val allOptions = options ++ Map("singleVariantColumn" -> "v")
+ checkAnswer(
+ spark.read.options(allOptions).csv(path.getCanonicalPath).selectExpr("cast(v as string)"),
+ expected.map(Row(_))
+ )
+ }
+
+ // A small partition size ensures each value is parsed independently and would otherwise get
+ // its type inferred.
+ withSQLConf(SQLConf.FILES_MAX_PARTITION_BYTES.key -> "10",
+ SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ // With variantRespectInferSchema on and inferSchema off, scalar values parse as string,
+ // while the null value (empty token) parses as a variant null. inferSchema defaults to
+ // false, so passing it explicitly and omitting it behave the same.
+ for (opts <- Seq(Map("variantRespectInferSchema" -> "true", "inferSchema" -> "false"),
+ Map("variantRespectInferSchema" -> "true"))) {
+ checkSingleVariant(opts,
+ """{"_c0":"field 1","_c1":"field2"}""",
+ """{"_c0":"100","_c1":"1.1"}""",
+ """{"_c0":"2000-01-01","_c1":"2000-01-01 01:02:03"}""",
+ """{"_c0":null,"_c1":"true"}""",
+ """{"_c0":"1e9","_c1":"hello","_c2":"extra"}""",
+ """{"_c0":"missing"}""")
+ }
+
+ // Explicitly enabling inference or disabling variantRespectInferSchema causes types
+ // to be inferred. Timestamps should be inferred correctly for all parsing modes.
+ for ((policy, timestampField) <- Seq(
+ "CORRECTED" -> "\"2000-01-01 01:02:03+00:00\"",
+ "LEGACY" -> "\"2000-01-01 01:02:03\"")) {
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_POLICY.key -> policy) {
+ // Explicitly enabling inference infers types even with variantRespectInferSchema on.
+ checkSingleVariant(Map("variantRespectInferSchema" -> "true", "inferSchema" -> "true"),
+ """{"_c0":"field 1","_c1":"field2"}""",
+ """{"_c0":100,"_c1":1.1}""",
+ s"""{"_c0":"2000-01-01","_c1":$timestampField}""",
+ """{"_c0":null,"_c1":true}""",
+ """{"_c0":1000000000,"_c1":"hello","_c2":"extra"}""",
+ """{"_c0":"missing"}""")
+
+ // variantRespectInferSchema defaults to false, so scalar types are inferred regardless
+ // of inferSchema.
+ checkSingleVariant(Map("inferSchema" -> "false"),
+ """{"_c0":"field 1","_c1":"field2"}""",
+ """{"_c0":100,"_c1":1.1}""",
+ s"""{"_c0":"2000-01-01","_c1":$timestampField}""",
+ """{"_c0":null,"_c1":true}""",
+ """{"_c0":1000000000,"_c1":"hello","_c2":"extra"}""",
+ """{"_c0":"missing"}""")
+ }
+ }
+ }
+
+ // Works with a header too: field names come from the header, values stay strings.
+ withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ checkSingleVariant(
+ Map("header" -> "true", "variantRespectInferSchema" -> "true", "inferSchema" -> "false"),
+ """{"field 1":"100","field2":"1.1"}""",
+ """{"field 1":"2000-01-01","field2":"2000-01-01 01:02:03"}""",
+ """{"field 1":null,"field2":"true"}""",
+ """{"field 1":"1e9","field2":"hello"}""",
+ """{"field 1":"missing"}""")
+ }
+
+ // Values whose textual form is lost by type inference: leading zeros ("0001" -> 1), a leading
+ // sign ("+5" -> 5), and boolean case ("True" -> true). With retention on they are kept
+ // verbatim as strings; with it off they are inferred (the surprising default behavior).
+ val lossyData =
+ """0001,+5
+ |True,007
+ |""".stripMargin
+ Files.write(path.toPath, lossyData.getBytes(StandardCharsets.UTF_8))
+
+ withSQLConf(SQLConf.FILES_MAX_PARTITION_BYTES.key -> "10") {
+ checkSingleVariant(Map("variantRespectInferSchema" -> "true", "inferSchema" -> "false"),
+ """{"_c0":"0001","_c1":"+5"}""",
+ """{"_c0":"True","_c1":"007"}""")
+
+ checkSingleVariant(Map("inferSchema" -> "false"),
+ """{"_c0":1,"_c1":5}""",
+ """{"_c0":true,"_c1":7}""")
+ }
+ }
+ }
+
+ test("csv variant retains explicit schema with variantRespectInferSchema") {
+ // Explicit schemas with VariantType columns share a converter with singleVariantColumn,
+ // so retention applies there too. Check and assert the type using schema_of_variant.
+ withTempPath { path =>
+ Files.write(path.toPath, "1000,0001,'0001'\n".getBytes(StandardCharsets.UTF_8))
+ val schema = "c1 variant, c2 variant, c3 variant"
+ val exprs = Seq("schema_of_variant(c1)", "schema_of_variant(c2)", "schema_of_variant(c3)")
+
+ // With variantRespectInferSchema values should be treated as string
+ val retained = spark.read.schema(schema)
+ .options(Map("inferSchema" -> "false", "variantRespectInferSchema" -> "true"))
+ .csv(path.getCanonicalPath)
+ checkAnswer(retained.selectExpr(exprs: _*), Row("STRING", "STRING", "STRING"))
+
+ // Without variantRespectInferSchema values should be inferred. Integral values are
+ // inferred as BIGINT by default.
+ val inferred = spark.read.schema(schema).csv(path.getCanonicalPath)
+ checkAnswer(inferred.selectExpr(exprs: _*), Row("BIGINT", "BIGINT", "STRING"))
+ }
+ }
+
test("write variant with csv is disallowed") {
checkError(
exception = intercept[AnalysisException] {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org