You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2026/09/04 16:20:03 UTC

(spark) branch master updated: [SPARK-58917][SQL] Add option to respect `inferSchema` when reading CSV as variant

This is an automated email from the ASF dual-hosted git repository.

cloud-fan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 3a0845a58810 [SPARK-58917][SQL] Add option to respect `inferSchema` when reading CSV as variant
3a0845a58810 is described below

commit 3a0845a5881055a4e480cdb0fcdcd060d04adb95
Author: Greg Hansen <gr...@databricks.com>
AuthorDate: Sat Sep 5 00:19:36 2026 +0800

    [SPARK-58917][SQL] Add option to respect `inferSchema` when reading CSV as variant
    
    ### What changes were proposed in this pull request?
    
    This PR adds a new CSV read option, `variantRespectInferSchema` (default False). When set to True and `inferSchema` is False, the CSV-to-Variant parser preserves scalar leaf values as strings instead of inferring their types. This applies wherever CSV values are read into a VARIANT:
    
    * When using the `singleVariantColumn` option
    * When reading explicitly-typed VariantType columns
    
    #### Decision notes
    * We use a per-read reader option to allow users to set this with `.option(...)` instead of as a global Spark config.
    * The default is False to preserve the existing best-effort type inference unless a user opts-in.
    
    ### Why are the changes needed?
    
    When reading CSV records as Variant, the parser infers scalar types regardless of `inferSchema`. Text can be changed by type inference (e.g. "0001" is stored as the integer 1, "+5" as 5, "True" as boolean true). There is no way to retain the original string values.
    
    See [JIRA issue](https://issues.apache.org/jira/browse/SPARK-58917) for more details.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes — a new read option variantRespectInferSchema. Default false, so no change for existing workloads. With it enabled:
    
    ```scala
    spark
      .read
      .option("singleVariantColumn", "v")
      .option("variantRespectInferSchema", "true")
      .csv(path)
    ```
    
    ### How was this patch tested?
    
    Added a unit test in CSVSuite which is passing on CSVv1Suite and CSVv2Suite.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 4.8)
    
    Closes #58175 from ghanse/csv-variant-scalar-as-string.
    
    Authored-by: Greg Hansen <gr...@databricks.com>
    Signed-off-by: Wenchen Fan <we...@databricks.com>
---
 docs/sql-data-sources-csv.md                       |  10 +-
 .../apache/spark/sql/catalyst/csv/CSVOptions.scala |   4 +
 .../spark/sql/catalyst/csv/UnivocityParser.scala   |   5 +
 .../org/apache/spark/sql/CsvFunctionsSuite.scala   |  41 +++++++
 .../sql/execution/datasources/csv/CSVSuite.scala   | 122 ++++++++++++++++++++-
 5 files changed, 179 insertions(+), 3 deletions(-)

diff --git a/docs/sql-data-sources-csv.md b/docs/sql-data-sources-csv.md
index bf63fe2be5d3..6fe3dbd83af7 100644
--- a/docs/sql-data-sources-csv.md
+++ b/docs/sql-data-sources-csv.md
@@ -111,7 +111,7 @@ Data source options of CSV can be set via:
   <tr>
     <td><code>inferSchema</code></td>
     <td>false</td>
-    <td>Infers the input schema automatically from data. It requires one extra pass over the data. CSV built-in functions ignore this option.</td>
+    <td>Infers the input schema automatically from data. It requires one extra pass over the data. CSV built-in functions ignore this option, except as noted under <code>variantRespectInferSchema</code>.</td>
     <td>read</td>
   </tr>
   <tr>
@@ -225,7 +225,13 @@ Data source options of CSV can be set via:
   <tr>
     <td><code>singleVariantColumn</code></td>
     <td>(none)</td>
-    <td>If specified, the entire CSV record is parsed and stored as a single column of <code>VariantType</code> with the given column name, instead of being split into individual fields.</td>
+    <td>If specified, the entire CSV record is parsed and stored as a single column of <code>VariantType</code> with the given column name, instead of being split into individual fields. By default, scalar values are type-inferred inside the variant on a best-effort basis (for example, <code>"0001"</code> may be inferred as the integer <code>1</code>). To preserve scalar values as strings, set <code>variantRespectInferSchema</code> to <code>true</code> and <code>inferSchema</code> to <co [...]
+    <td>read</td>
+  </tr>
+  <tr>
+    <td><code>variantRespectInferSchema</code></td>
+    <td>false</td>
+    <td>When reading into a <code>VARIANT</code> (see <code>singleVariantColumn</code> or a <code>VariantType</code> column in the schema), controls whether the <code>inferSchema</code> option is honored. When <code>true</code> and <code>inferSchema</code> is <code>false</code>, scalar values are preserved as strings inside the variant instead of being inferred as boolean, long, decimal, date, or timestamp. When <code>false</code> (the default), scalar types are inferred on a best-effort [...]
     <td>read</td>
   </tr>
   <tr>
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
index 7db03a8a2323..a75a2da32f13 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
@@ -337,6 +337,9 @@ class CSVOptions(
   // E.g. spark.read.format("csv").option("singleVariantColumn", "colName")
   val singleVariantColumn: Option[String] = parameters.get(SINGLE_VARIANT_COLUMN)
 
+  // When true, reading CSV values into a VARIANT honors `inferSchema` (default: false).
+  val variantRespectInferSchema = getBool(VARIANT_RESPECT_INFER_SCHEMA, default = false)
+
   def needHeaderForSingleVariantColumn: Boolean =
     singleVariantColumn.isDefined && headerFlag
 
@@ -443,6 +446,7 @@ object CSVOptions extends DataSourceOptions {
   newOption(SEP, DELIMITER)
   val COLUMN_PRUNING = newOption("columnPruning")
   val SINGLE_VARIANT_COLUMN = newOption(DataSourceOptions.SINGLE_VARIANT_COLUMN)
+  val VARIANT_RESPECT_INFER_SCHEMA = newOption("variantRespectInferSchema")
 
   // Max error content length in CSV parser/writer exception messages, and the bound on the bad
   // record embedded in MALFORMED_CSV_RECORD errors.
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
index fa6bd19064f0..4583c328239e 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
@@ -481,6 +481,11 @@ class UnivocityParser(
         return
       }
 
+      if (!options.inferSchemaFlag && options.variantRespectInferSchema) {
+        builder.appendString(s)
+        return
+      }
+
       def parseLong(): DataType = {
         try {
           builder.appendLong(s.toLong)
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
index 455434b4de01..f79a7535cc14 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
@@ -869,6 +869,47 @@ class CsvFunctionsSuite extends SharedSparkSession {
       Seq(Row(s"""{null, $largeInput}""")))
   }
 
+  test("from_csv with variant: variantRespectInferSchema controls scalar inference") {
+    // from_csv shares the CSV parser, so it observes inferSchema only when
+    // variantRespectInferSchema is enabled: with inferSchema off the variant scalar stays a
+    // string, otherwise it is inferred. Without the flag, from_csv ignores inferSchema entirely.
+    val df = Seq("100").toDF("value")
+
+    // With variantRespectInferSchema enabled and inferSchema off the scalar is preserved as a
+    // string. inferSchema defaults to false, so setting the flag alone behaves the same.
+    for (options <- Seq(
+        Map("singleVariantColumn" -> "v", "variantRespectInferSchema" -> "true",
+          "inferSchema" -> "false"),
+        Map("singleVariantColumn" -> "v", "variantRespectInferSchema" -> "true"))) {
+      checkAnswer(
+        df.select(from_csv($"value", StructType.fromDDL("v variant"), options).cast("string")),
+        Seq(Row("""{{"_c0":"100"}}""")))
+    }
+
+    // Without variantRespectInferSchema, the scalar is inferred regardless of inferSchema.
+    // Both the default and explicitly configured inferSchema setting should result in inferred
+    // scalar values. Values should also be inferred when both options are enabled.
+    for (options <- Seq(
+        Map("singleVariantColumn" -> "v"),
+        Map("singleVariantColumn" -> "v", "inferSchema" -> "true"),
+        Map("singleVariantColumn" -> "v", "variantRespectInferSchema" -> "true",
+          "inferSchema" -> "true"))) {
+      checkAnswer(
+        df.select(from_csv($"value", StructType.fromDDL("v variant"), options).cast("string")),
+        Seq(Row("""{{"_c0":100}}""")))
+    }
+
+    // Explicit VariantType columns should behave the same. schema_of_variant tells the string
+    // "100" from the inferred long.
+    checkAnswer(
+      df.selectExpr("schema_of_variant(from_csv(value, 'a variant', " +
+        "map('variantRespectInferSchema', 'true', 'inferSchema', 'false')).a)"),
+      Seq(Row("STRING")))
+    checkAnswer(
+      df.selectExpr("schema_of_variant(from_csv(value, 'a variant').a)"),
+      Seq(Row("BIGINT")))
+  }
+
   test("from_csv with variant: extreme negative scale decimal does not hang") {
     // A value like "1E99999" parses to a BigDecimal with scale=-99999.
     // Calling setScale(0) on it would hang, so it should fall through to string.
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
index 3aefe25f6a08..0d0249fbad54 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
@@ -3342,7 +3342,7 @@ abstract class CSVSuite
   }
 
   test("validate CSV Options") {
-    assert(CSVOptions.getAllOptions.size == 42)
+    assert(CSVOptions.getAllOptions.size == 43)
     // Please add validation on any new CSV options here
     assert(CSVOptions.isValidOption("header"))
     assert(CSVOptions.isValidOption("inferSchema"))
@@ -3386,6 +3386,7 @@ abstract class CSVSuite
     assert(CSVOptions.isValidOption("delimiter"))
     assert(CSVOptions.isValidOption("singleVariantColumn"))
     assert(CSVOptions.isValidOption("columnPruning"))
+    assert(CSVOptions.isValidOption("variantRespectInferSchema"))
     // Please add validation on any new CSV options with alternative here
     assert(CSVOptions.getAlternativeOption("sep").contains("delimiter"))
     assert(CSVOptions.getAlternativeOption("delimiter").contains("sep"))
@@ -3898,6 +3899,125 @@ abstract class CSVSuite
     }
   }
 
+  test("csv variant retains scalars as strings when inferSchema is disabled") {
+    withTempPath { path =>
+      val data =
+        """field 1,field2
+          |100,1.1
+          |2000-01-01,2000-01-01 01:02:03
+          |,true
+          |1e9,hello,extra
+          |missing
+          |""".stripMargin
+      Files.write(path.toPath, data.getBytes(StandardCharsets.UTF_8))
+
+      def checkSingleVariant(options: Map[String, String], expected: String*): Unit = {
+        val allOptions = options ++ Map("singleVariantColumn" -> "v")
+        checkAnswer(
+          spark.read.options(allOptions).csv(path.getCanonicalPath).selectExpr("cast(v as string)"),
+          expected.map(Row(_))
+        )
+      }
+
+      // A small partition size ensures each value is parsed independently and would otherwise get
+      // its type inferred.
+      withSQLConf(SQLConf.FILES_MAX_PARTITION_BYTES.key -> "10",
+        SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+        // With variantRespectInferSchema on and inferSchema off, scalar values parse as string,
+        // while the null value (empty token) parses as a variant null. inferSchema defaults to
+        // false, so passing it explicitly and omitting it behave the same.
+        for (opts <- Seq(Map("variantRespectInferSchema" -> "true", "inferSchema" -> "false"),
+            Map("variantRespectInferSchema" -> "true"))) {
+          checkSingleVariant(opts,
+            """{"_c0":"field 1","_c1":"field2"}""",
+            """{"_c0":"100","_c1":"1.1"}""",
+            """{"_c0":"2000-01-01","_c1":"2000-01-01 01:02:03"}""",
+            """{"_c0":null,"_c1":"true"}""",
+            """{"_c0":"1e9","_c1":"hello","_c2":"extra"}""",
+            """{"_c0":"missing"}""")
+        }
+
+        // Explicitly enabling inference or disabling variantRespectInferSchema causes types
+        // to be inferred. Timestamps should be inferred correctly for all parsing modes.
+        for ((policy, timestampField) <- Seq(
+            "CORRECTED" -> "\"2000-01-01 01:02:03+00:00\"",
+            "LEGACY" -> "\"2000-01-01 01:02:03\"")) {
+          withSQLConf(SQLConf.LEGACY_TIME_PARSER_POLICY.key -> policy) {
+            // Explicitly enabling inference infers types even with variantRespectInferSchema on.
+            checkSingleVariant(Map("variantRespectInferSchema" -> "true", "inferSchema" -> "true"),
+              """{"_c0":"field 1","_c1":"field2"}""",
+              """{"_c0":100,"_c1":1.1}""",
+              s"""{"_c0":"2000-01-01","_c1":$timestampField}""",
+              """{"_c0":null,"_c1":true}""",
+              """{"_c0":1000000000,"_c1":"hello","_c2":"extra"}""",
+              """{"_c0":"missing"}""")
+
+            // variantRespectInferSchema defaults to false, so scalar types are inferred regardless
+            // of inferSchema.
+            checkSingleVariant(Map("inferSchema" -> "false"),
+              """{"_c0":"field 1","_c1":"field2"}""",
+              """{"_c0":100,"_c1":1.1}""",
+              s"""{"_c0":"2000-01-01","_c1":$timestampField}""",
+              """{"_c0":null,"_c1":true}""",
+              """{"_c0":1000000000,"_c1":"hello","_c2":"extra"}""",
+              """{"_c0":"missing"}""")
+          }
+        }
+      }
+
+      // Works with a header too: field names come from the header, values stay strings.
+      withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+        checkSingleVariant(
+          Map("header" -> "true", "variantRespectInferSchema" -> "true", "inferSchema" -> "false"),
+          """{"field 1":"100","field2":"1.1"}""",
+          """{"field 1":"2000-01-01","field2":"2000-01-01 01:02:03"}""",
+          """{"field 1":null,"field2":"true"}""",
+          """{"field 1":"1e9","field2":"hello"}""",
+          """{"field 1":"missing"}""")
+      }
+
+      // Values whose textual form is lost by type inference: leading zeros ("0001" -> 1), a leading
+      // sign ("+5" -> 5), and boolean case ("True" -> true). With retention on they are kept
+      // verbatim as strings; with it off they are inferred (the surprising default behavior).
+      val lossyData =
+        """0001,+5
+          |True,007
+          |""".stripMargin
+      Files.write(path.toPath, lossyData.getBytes(StandardCharsets.UTF_8))
+
+      withSQLConf(SQLConf.FILES_MAX_PARTITION_BYTES.key -> "10") {
+        checkSingleVariant(Map("variantRespectInferSchema" -> "true", "inferSchema" -> "false"),
+          """{"_c0":"0001","_c1":"+5"}""",
+          """{"_c0":"True","_c1":"007"}""")
+
+        checkSingleVariant(Map("inferSchema" -> "false"),
+          """{"_c0":1,"_c1":5}""",
+          """{"_c0":true,"_c1":7}""")
+      }
+    }
+  }
+
+  test("csv variant retains explicit schema with variantRespectInferSchema") {
+    // Explicit schemas with VariantType columns share a converter with singleVariantColumn,
+    // so retention applies there too. Check and assert the type using schema_of_variant.
+    withTempPath { path =>
+      Files.write(path.toPath, "1000,0001,'0001'\n".getBytes(StandardCharsets.UTF_8))
+      val schema = "c1 variant, c2 variant, c3 variant"
+      val exprs = Seq("schema_of_variant(c1)", "schema_of_variant(c2)", "schema_of_variant(c3)")
+
+      // With variantRespectInferSchema values should be treated as string
+      val retained = spark.read.schema(schema)
+        .options(Map("inferSchema" -> "false", "variantRespectInferSchema" -> "true"))
+        .csv(path.getCanonicalPath)
+      checkAnswer(retained.selectExpr(exprs: _*), Row("STRING", "STRING", "STRING"))
+
+      // Without variantRespectInferSchema values should be inferred. Integral values are
+      // inferred as BIGINT by default.
+      val inferred = spark.read.schema(schema).csv(path.getCanonicalPath)
+      checkAnswer(inferred.selectExpr(exprs: _*), Row("BIGINT", "BIGINT", "STRING"))
+    }
+  }
+
   test("write variant with csv is disallowed") {
     checkError(
       exception = intercept[AnalysisException] {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org