You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2026/09/14 05:39:12 UTC
(spark) branch branch-4.x updated: [SPARK-58946][SQL] Fix CSV file extension validations
This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new c1c9f33c64f7 [SPARK-58946][SQL] Fix CSV file extension validations
c1c9f33c64f7 is described below
commit c1c9f33c64f79b9fd1678d356abd1bdbe1f6aa58
Author: Subhramit Basu <su...@live.in>
AuthorDate: Mon Sep 14 13:38:45 2026 +0800
[SPARK-58946][SQL] Fix CSV file extension validations
### What changes were proposed in this pull request?
**Note: This depends on SPARK-58945 (PR https://github.com/apache/spark/pull/58225).** Until that is merged this path surfaces `INTERNAL_ERROR` rather than the intended error, so the new tests will not pass on its own.
Change the `extension` option validation in `CSVOptions.scala:127` from:
```scala
ext.size != 3 && !ext.forall(_.isLetter)
```
to:
```scala
ext.isEmpty || !ext.forall(_.isLetter)
```
It also updates the `INVALID_PARAMETER_VALUE.EXTENSION` message in `error-conditions.json` to match the new rule, and adds regression coverage in `CSVSuite`.
### Why are the changes needed?
On `master`, the current check rejects only values that are **both** not three characters long and contain a non-letter. That means values such as these are accepted today:
| value | why it should be rejected |
|---|---|
| `` | empty suffix |
| `ab1` | contains a non-letter |
| `777` | contains only numbers |
| `a/b` | contains a path separator |
Example repro:
```scala
val path = "/tmp/spark58946-demo"
spark.range(1).write.mode("overwrite").option("extension", "a/b").csv(path)
```
At the same time, alphabetic extensions of arbitrary length such as `a` and `abcd` are already accepted on master.
The value flows into the output filename (`CSVWrite.scala:61`, `CSVFileFormat.scala:91`), so the technical requirement here is that the suffix be non-empty and filename-safe. The `forall(_.isLetter)` part is what enforces that. The exact-three-character restriction does not add safety, and the current `&&` check still lets through the actually problematic cases above.
We keep the letters-only validation, explicitly rejects the empty case, and updates the error text to describe that behavior accurately.
### Does this PR introduce _any_ user-facing change?
Yes. CSV writes that currently succeed with invalid extensions such as `""`, `ab1`, or `a/b` now fail with `INVALID_PARAMETER_VALUE.EXTENSION`.
Alphabetic extensions of arbitrary length, such as `a`, `abcd`, and `tsv`, remain valid.
### How was this patch tested?
#### 1. Performed manual repro -
- **On `master.904b993`:**
<img width="1101" height="490" alt="image" src="https://github.com/user-attachments/assets/f4e8eedb-ad6b-464d-9f38-d0798b68e782" />
- **On this branch:**
<img width="1107" height="553" alt="image" src="https://github.com/user-attachments/assets/3675dc4e-05a7-4363-b150-cfc5d941ffa3" />
#### 2. Added/updated `CSVSuite` coverage to:
- reject `""`, `ab1`, and `a/b`
- accept alphabetic extensions of arbitrary length such as `a` and `abcd`
The existing `SPARK-50616` `tsv` test remains valid under the new rule.
Full suites to run:
```bash
build/sbt "sql/testOnly *CSVv1Suite"
build/sbt "sql/testOnly *CSVv2Suite"
build/sbt "sql/testOnly *CSVLegacyTimeParserSuite"
```
should pass once https://github.com/apache/spark/pull/58225 is merged and synced here.
### Was this patch authored or co-authored using generative AI tooling?
Assisted-by: Claude Opus 5
Closes #58226 from subhramit/SPARK-58946.
Authored-by: Subhramit Basu <su...@live.in>
Signed-off-by: yangjie01 <ya...@baidu.com>
(cherry picked from commit 7703208beb84bdb504cd53426565a07a07436250)
Signed-off-by: yangjie01 <ya...@baidu.com>
---
.../src/main/resources/error/error-conditions.json | 2 +-
docs/sql-data-sources-csv.md | 2 +-
docs/sql-migration-guide.md | 1 +
.../apache/spark/sql/catalyst/csv/CSVOptions.scala | 2 +-
.../sql/execution/datasources/csv/CSVSuite.scala | 35 ++++++++++++++++++++++
5 files changed, 39 insertions(+), 3 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json
index c98d641060da..798ecbace0bc 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -4875,7 +4875,7 @@
},
"EXTENSION" : {
"message" : [
- "Invalid extension: <invalidValue>. Extension is limited to exactly 3 letters (e.g. csv, tsv, etc...)"
+ "Invalid extension: <invalidValue>. Extension must be non-empty and contain only letters."
]
},
"HMAC_ALGORITHM" : {
diff --git a/docs/sql-data-sources-csv.md b/docs/sql-data-sources-csv.md
index 7063685fdfed..7408007e3247 100644
--- a/docs/sql-data-sources-csv.md
+++ b/docs/sql-data-sources-csv.md
@@ -63,7 +63,7 @@ Data source options of CSV can be set via:
<tr>
<td><code>extension</code></td>
<td>csv</td>
- <td>Sets the file extension for the output files. Limited to letters. Length must equal 3.</td>
+ <td>Sets the file extension for the output files. Must be non-empty and contain only letters. If it matches a compression codec suffix (for example, <code>gz</code> or <code>zst</code>), set <code>compression</code> too; otherwise Spark may try to decompress plain-text output when reading it back.</td>
<td>write</td>
</tr>
<tr>
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index b6dfcc34891a..7f834591f362 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -28,6 +28,7 @@ license: |
- Since Spark 4.4, the Oracle JDBC datasource maps Oracle DATE and TIMESTAMP to TimestampNTZType unconditionally, ignoring the JDBC read option `preferTimestampNTZ` (DATE only when the driver default `oracle.jdbc.mapDateToTimestamp=true` surfaces it as TIMESTAMP; otherwise DATE stays DateType). In 4.3 and earlier these were read as TimestampType, or as TimestampNTZType only when `preferTimestampNTZ=true`. This faithfully represents these zoneless Oracle types. Unaffected: Oracle DATE rea [...]
- Since Spark 4.4, a nondeterministic common expression that a conditional branch reads more than once is evaluated once, and every read sees that one value. `BETWEEN` and `NULLIF` read their input twice, so an expression such as `CASE WHEN c THEN monotonically_increasing_id() BETWEEN 3 AND 5 END` previously gave each read its own value and could return a different result. Queries whose results depended on the earlier behaviour change, and an optimized or physical plan may now show a `wi [...]
- Since Spark 4.4, for storage-partitioned joins, `spark.sql.requireAllClusterKeysForCoPartition` requires every join key to be covered by some partition key instead of matching the partition keys positionally. As a result, a join-key column partitioned by more than one transform no longer prevents shuffle elimination, and `spark.sql.sources.v2.bucketing.allowKeysSubsetOfPartitionKeys.enabled` no longer additionally requires `spark.sql.requireAllClusterKeysForCoPartition` to be `false` w [...]
+- Since Spark 4.4, the CSV `extension` option must be non-empty and contain only letters. In Spark 4.3 and earlier, due to a bug in the extension validation check, empty strings and three-character suffixes containing non-letters (for example `ab1`) were accepted. This affects both writes and reads of CSV tables whose persisted `OPTIONS` contain such values; update the option to a non-empty letters-only suffix to restore access.
- Since Spark 4.4, when `array_repeat` or `array_insert` is asked to build an array larger than the maximum supported array length, generated code raises the same error as interpreted evaluation. `array_repeat` now fails with `COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER` instead of the internal error `_LEGACY_ERROR_TEMP_2176`, and `array_insert` fails with `COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION` instead of `COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER`, which named a `count` parameter that `a [...]
## Upgrading from Spark SQL 4.2 to 4.3
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
index 359b28c261eb..e628192efe4a 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
@@ -124,7 +124,7 @@ class CSVOptions(
val extension = {
val ext = parameters.getOrElse(EXTENSION, "csv")
- if (ext.size != 3 && !ext.forall(_.isLetter)) {
+ if (ext.isEmpty || !ext.forall(_.isLetter)) {
throw QueryExecutionErrors.invalidFileExtensionError("csv", ext)
}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
index c698b351718e..b34b13efc512 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
@@ -3169,6 +3169,41 @@ abstract class CSVSuite
}
}
+ test("SPARK-58946: reject empty or non-letter file extensions") {
+ Seq("", "ab1", "a/b").foreach { ext =>
+ withTempPath { path =>
+ checkError(
+ exception = intercept[SparkIllegalArgumentException] {
+ spark.range(1).write.option("extension", ext).csv(path.getAbsolutePath)
+ },
+ condition = "INVALID_PARAMETER_VALUE.EXTENSION",
+ parameters = Map(
+ "functionName" -> "`csv`",
+ "parameter" -> "`extension`",
+ "invalidValue" -> s"`$ext`"))
+ }
+ }
+ }
+
+ test("SPARK-58946: allow alphabetic file extensions of arbitrary length") {
+ Seq("a", "abcd").foreach { ext =>
+ withTempPath { path =>
+ val input = Seq(
+ "1423-11-12T23:41:00",
+ "1765-03-28",
+ "2016-01-28T20:00:00"
+ ).toDF().repartition(1)
+ input.write.option("extension", ext).csv(path.getAbsolutePath)
+
+ val files = Files.list(path.toPath)
+ .iterator().asScala.map(_.getFileName.toString)
+ .toList.filter(_.endsWith(s".$ext"))
+
+ assert(files.size == 1)
+ }
+ }
+ }
+
test("SPARK-50616: We can write with a tsv file extension") {
withTempPath { path =>
val input = Seq(
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org