You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/10/22 15:06:36 UTC
(spark-connect-swift) branch main updated: [SPARK-60471] Fix `DataFrame.withColumnRenamed` to not crash on duplicate column names
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 6ab8a43 [SPARK-60471] Fix `DataFrame.withColumnRenamed` to not crash on duplicate column names
6ab8a43 is described below
commit 6ab8a4342e04666275914a7911dc59bd8b317eba
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Thu Oct 22 08:06:29 2026 -0700
[SPARK-60471] Fix `DataFrame.withColumnRenamed` to not crash on duplicate column names
### What changes were proposed in this pull request?
This PR aims to fix `DataFrame.withColumnRenamed(_ colNames: [String], _ newColNames: [String])` to not crash on duplicate column names. It now sends the renames in the given order via the `renames` field of `WithColumnsRenamed`, like the Scala and Python Spark Connect clients.
- Add `SparkConnectClient.getWithColumnRenamed(_:_:_:)`, which builds `WithColumnsRenamed.renames` from the given column name lists.
- `withColumnRenamed(_ colNames: [String], _ newColNames: [String])` uses it instead of `Dictionary(uniqueKeysWithValues:)` and the deprecated `rename_columns_map` field. If the lists have different lengths, the extra names are ignored as before.
`withColumnRenamed(_ existingName: String, _ newName: String)` and `withColumnRenamed(_ colsMap: [String: String])` are not changed.
### Why are the changes needed?
`Dictionary(uniqueKeysWithValues:)` traps on duplicate keys, so the following crashes the client process although the API does not throw.
```swift
// Fatal error: Duplicate values for key: 'a'
df.withColumnRenamed(["a", "a"], ["x", "y"])
```
In addition, `rename_columns_map` is an unordered `map`, and `swift-protobuf` serializes it in the iteration order of `Dictionary`, which differs across processes. Since the server applies the renames one by one, the result of order-sensitive renames was not deterministic. For example, `df.withColumnRenamed(["a", "x"], ["x", "y"])` returned `["x", "b", "c", "d"]` or `["y", "b", "c", "d"]` depending on the process.
The `renames` field was added by SPARK-46261 in Apache Spark [v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19), so all servers supported by this client accept it. In `v4.2.0` (2026-07-11), the server reads it in [SparkConnectPlanner](https://github.com/apache/spark/blob/v4.2.0/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L1287-L1310) and applies the renames in order in [UnresolvedStarWithColumnsRenames](https [...]
### Does this PR introduce _any_ user-facing change?
Yes, it fixes a bug. Duplicate column names no longer crash, and the renames are applied in the given order, the same as the Scala and Python clients. The API signatures do not change.
- `df.withColumnRenamed(["a", "a"], ["x", "y"])`: crash → `["x", "b", "c", "d"]`. The second `a` is ignored because `a` is already renamed to `x`, like `df.withColumnRenamed("a", "x").withColumnRenamed("a", "y")`.
- `df.withColumnRenamed(["a", "x"], ["x", "y"])`: `["x", "b", "c", "d"]` or `["y", "b", "c", "d"]` → always `["y", "b", "c", "d"]`.
### How was this patch tested?
Pass the CIs with the newly added test cases.
- `DataFrameInternalTests.withColumnRenamedPlan`, which only inspects the generated plan, so it does not require a running server.
- `DataFrameTests.withColumnRenamedWithDuplicateNames`
Both crashed with `Fatal error: Duplicate values for key: 'a'` before this fix. In addition, `DataFrameTests` and `DataFrameInternalTests` passed against `apache/spark:4.2.0` and `apache/spark:4.0.4-java21-scala` Spark Connect servers, and the Scala Spark Connect client returned the same results against `apache/spark:4.2.0`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #617 from dongjoon-hyun/SPARK-60471.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
Sources/SparkConnect/DataFrame+Transformations.swift | 4 ++--
Sources/SparkConnect/SparkConnectClient.swift | 14 ++++++++++++++
Tests/SparkConnectTests/DataFrameInternalTests.swift | 13 +++++++++++++
Tests/SparkConnectTests/DataFrameTests.swift | 10 ++++++++++
4 files changed, 39 insertions(+), 2 deletions(-)
diff --git a/Sources/SparkConnect/DataFrame+Transformations.swift b/Sources/SparkConnect/DataFrame+Transformations.swift
index e40e25f..14a2973 100644
--- a/Sources/SparkConnect/DataFrame+Transformations.swift
+++ b/Sources/SparkConnect/DataFrame+Transformations.swift
@@ -287,9 +287,9 @@ extension DataFrame {
/// - newColNames: A list of new column names.
/// - Returns: A ``DataFrame`` with the renamed columns.
public func withColumnRenamed(_ colNames: [String], _ newColNames: [String]) -> DataFrame {
- let dic = Dictionary(uniqueKeysWithValues: zip(colNames, newColNames))
return DataFrame(
- spark: self.spark, plan: SparkConnectClient.getWithColumnRenamed(self.plan.root, dic))
+ spark: self.spark,
+ plan: SparkConnectClient.getWithColumnRenamed(self.plan.root, colNames, newColNames))
}
/// Returns a new Dataset with columns renamed. This is a no-op if schema doesn't contain existingName.
diff --git a/Sources/SparkConnect/SparkConnectClient.swift b/Sources/SparkConnect/SparkConnectClient.swift
index 0822cd9..4bbbfb0 100644
--- a/Sources/SparkConnect/SparkConnectClient.swift
+++ b/Sources/SparkConnect/SparkConnectClient.swift
@@ -748,6 +748,20 @@ public actor SparkConnectClient {
return createPlan { $0.withColumnsRenamed = withColumnsRenamed }
}
+ static func getWithColumnRenamed(
+ _ child: Relation, _ colNames: [String], _ newColNames: [String]
+ ) -> Plan {
+ var withColumnsRenamed = WithColumnsRenamed()
+ withColumnsRenamed.input = child
+ withColumnsRenamed.renames = zip(colNames, newColNames).map { (colName, newColName) in
+ var rename = WithColumnsRenamed.Rename()
+ rename.colName = colName
+ rename.newColName = newColName
+ return rename
+ }
+ return createPlan { $0.withColumnsRenamed = withColumnsRenamed }
+ }
+
static func getWithColumns(_ child: Relation, _ colsMap: [String: String]) -> Plan {
var withColumns = WithColumns()
withColumns.input = child
diff --git a/Tests/SparkConnectTests/DataFrameInternalTests.swift b/Tests/SparkConnectTests/DataFrameInternalTests.swift
index 2b54315..ee091ec 100644
--- a/Tests/SparkConnectTests/DataFrameInternalTests.swift
+++ b/Tests/SparkConnectTests/DataFrameInternalTests.swift
@@ -236,4 +236,17 @@ struct DataFrameInternalTests {
let other = await df.stat.sampleBy("key", [[1]: 0.5], 0).plan.root.sampleBy.fractions
#expect(other[0].stratum.literalType == .string("[1]"))
}
+
+ @Test
+ func withColumnRenamedPlan() async throws {
+ // This test doesn't require a running server because it only inspects the plan.
+ let spark = try SparkSession("sc://localhost")
+ var plan = Plan()
+ plan.root.range.end = 10
+ let df = DataFrame(spark: spark, plan: plan)
+ let renames = await df.withColumnRenamed(["a", "b", "a"], ["b", "c", "d"]).plan.root
+ .withColumnsRenamed.renames
+ #expect(renames.map { $0.colName } == ["a", "b", "a"])
+ #expect(renames.map { $0.newColName } == ["b", "c", "d"])
+ }
}
diff --git a/Tests/SparkConnectTests/DataFrameTests.swift b/Tests/SparkConnectTests/DataFrameTests.swift
index 063a359..8687a1b 100644
--- a/Tests/SparkConnectTests/DataFrameTests.swift
+++ b/Tests/SparkConnectTests/DataFrameTests.swift
@@ -635,6 +635,16 @@ struct DataFrameTests {
await spark.stop()
}
+ @Test
+ func withColumnRenamedWithDuplicateNames() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let df = try await spark.sql("SELECT 1 a, 2 b, 3 c, 4 d")
+ // Renames are applied in the given order. The second `a` is ignored because `a` is renamed.
+ #expect(try await df.withColumnRenamed(["a", "a"], ["x", "y"]).columns == ["x", "b", "c", "d"])
+ #expect(try await df.withColumnRenamed(["a", "x"], ["x", "y"]).columns == ["y", "b", "c", "d"])
+ await spark.stop()
+ }
+
@Test
func withColumn() async throws {
let spark = try await SparkSession.builder.getOrCreate()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org