You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/10/22 15:06:36 UTC

(spark-connect-swift) branch main updated: [SPARK-60471] Fix `DataFrame.withColumnRenamed` to not crash on duplicate column names

This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git


The following commit(s) were added to refs/heads/main by this push:
     new 6ab8a43  [SPARK-60471] Fix `DataFrame.withColumnRenamed` to not crash on duplicate column names
6ab8a43 is described below

commit 6ab8a4342e04666275914a7911dc59bd8b317eba
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Thu Oct 22 08:06:29 2026 -0700

    [SPARK-60471] Fix `DataFrame.withColumnRenamed` to not crash on duplicate column names
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix `DataFrame.withColumnRenamed(_ colNames: [String], _ newColNames: [String])` to not crash on duplicate column names. It now sends the renames in the given order via the `renames` field of `WithColumnsRenamed`, like the Scala and Python Spark Connect clients.
    
    - Add `SparkConnectClient.getWithColumnRenamed(_:_:_:)`, which builds `WithColumnsRenamed.renames` from the given column name lists.
    - `withColumnRenamed(_ colNames: [String], _ newColNames: [String])` uses it instead of `Dictionary(uniqueKeysWithValues:)` and the deprecated `rename_columns_map` field. If the lists have different lengths, the extra names are ignored as before.
    
    `withColumnRenamed(_ existingName: String, _ newName: String)` and `withColumnRenamed(_ colsMap: [String: String])` are not changed.
    
    ### Why are the changes needed?
    
    `Dictionary(uniqueKeysWithValues:)` traps on duplicate keys, so the following crashes the client process although the API does not throw.
    
    ```swift
    // Fatal error: Duplicate values for key: 'a'
    df.withColumnRenamed(["a", "a"], ["x", "y"])
    ```
    
    In addition, `rename_columns_map` is an unordered `map`, and `swift-protobuf` serializes it in the iteration order of `Dictionary`, which differs across processes. Since the server applies the renames one by one, the result of order-sensitive renames was not deterministic. For example, `df.withColumnRenamed(["a", "x"], ["x", "y"])` returned `["x", "b", "c", "d"]` or `["y", "b", "c", "d"]` depending on the process.
    
    The `renames` field was added by SPARK-46261 in Apache Spark [v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19), so all servers supported by this client accept it. In `v4.2.0` (2026-07-11), the server reads it in [SparkConnectPlanner](https://github.com/apache/spark/blob/v4.2.0/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L1287-L1310) and applies the renames in order in [UnresolvedStarWithColumnsRenames](https [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, it fixes a bug. Duplicate column names no longer crash, and the renames are applied in the given order, the same as the Scala and Python clients. The API signatures do not change.
    
    - `df.withColumnRenamed(["a", "a"], ["x", "y"])`: crash → `["x", "b", "c", "d"]`. The second `a` is ignored because `a` is already renamed to `x`, like `df.withColumnRenamed("a", "x").withColumnRenamed("a", "y")`.
    - `df.withColumnRenamed(["a", "x"], ["x", "y"])`: `["x", "b", "c", "d"]` or `["y", "b", "c", "d"]` → always `["y", "b", "c", "d"]`.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases.
    
    - `DataFrameInternalTests.withColumnRenamedPlan`, which only inspects the generated plan, so it does not require a running server.
    - `DataFrameTests.withColumnRenamedWithDuplicateNames`
    
    Both crashed with `Fatal error: Duplicate values for key: 'a'` before this fix. In addition, `DataFrameTests` and `DataFrameInternalTests` passed against `apache/spark:4.2.0` and `apache/spark:4.0.4-java21-scala` Spark Connect servers, and the Scala Spark Connect client returned the same results against `apache/spark:4.2.0`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #617 from dongjoon-hyun/SPARK-60471.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 Sources/SparkConnect/DataFrame+Transformations.swift |  4 ++--
 Sources/SparkConnect/SparkConnectClient.swift        | 14 ++++++++++++++
 Tests/SparkConnectTests/DataFrameInternalTests.swift | 13 +++++++++++++
 Tests/SparkConnectTests/DataFrameTests.swift         | 10 ++++++++++
 4 files changed, 39 insertions(+), 2 deletions(-)

diff --git a/Sources/SparkConnect/DataFrame+Transformations.swift b/Sources/SparkConnect/DataFrame+Transformations.swift
index e40e25f..14a2973 100644
--- a/Sources/SparkConnect/DataFrame+Transformations.swift
+++ b/Sources/SparkConnect/DataFrame+Transformations.swift
@@ -287,9 +287,9 @@ extension DataFrame {
   ///   - newColNames: A list of new column names.
   /// - Returns: A ``DataFrame`` with the renamed columns.
   public func withColumnRenamed(_ colNames: [String], _ newColNames: [String]) -> DataFrame {
-    let dic = Dictionary(uniqueKeysWithValues: zip(colNames, newColNames))
     return DataFrame(
-      spark: self.spark, plan: SparkConnectClient.getWithColumnRenamed(self.plan.root, dic))
+      spark: self.spark,
+      plan: SparkConnectClient.getWithColumnRenamed(self.plan.root, colNames, newColNames))
   }
 
   /// Returns a new Dataset with columns renamed. This is a no-op if schema doesn't contain existingName.
diff --git a/Sources/SparkConnect/SparkConnectClient.swift b/Sources/SparkConnect/SparkConnectClient.swift
index 0822cd9..4bbbfb0 100644
--- a/Sources/SparkConnect/SparkConnectClient.swift
+++ b/Sources/SparkConnect/SparkConnectClient.swift
@@ -748,6 +748,20 @@ public actor SparkConnectClient {
     return createPlan { $0.withColumnsRenamed = withColumnsRenamed }
   }
 
+  static func getWithColumnRenamed(
+    _ child: Relation, _ colNames: [String], _ newColNames: [String]
+  ) -> Plan {
+    var withColumnsRenamed = WithColumnsRenamed()
+    withColumnsRenamed.input = child
+    withColumnsRenamed.renames = zip(colNames, newColNames).map { (colName, newColName) in
+      var rename = WithColumnsRenamed.Rename()
+      rename.colName = colName
+      rename.newColName = newColName
+      return rename
+    }
+    return createPlan { $0.withColumnsRenamed = withColumnsRenamed }
+  }
+
   static func getWithColumns(_ child: Relation, _ colsMap: [String: String]) -> Plan {
     var withColumns = WithColumns()
     withColumns.input = child
diff --git a/Tests/SparkConnectTests/DataFrameInternalTests.swift b/Tests/SparkConnectTests/DataFrameInternalTests.swift
index 2b54315..ee091ec 100644
--- a/Tests/SparkConnectTests/DataFrameInternalTests.swift
+++ b/Tests/SparkConnectTests/DataFrameInternalTests.swift
@@ -236,4 +236,17 @@ struct DataFrameInternalTests {
     let other = await df.stat.sampleBy("key", [[1]: 0.5], 0).plan.root.sampleBy.fractions
     #expect(other[0].stratum.literalType == .string("[1]"))
   }
+
+  @Test
+  func withColumnRenamedPlan() async throws {
+    // This test doesn't require a running server because it only inspects the plan.
+    let spark = try SparkSession("sc://localhost")
+    var plan = Plan()
+    plan.root.range.end = 10
+    let df = DataFrame(spark: spark, plan: plan)
+    let renames = await df.withColumnRenamed(["a", "b", "a"], ["b", "c", "d"]).plan.root
+      .withColumnsRenamed.renames
+    #expect(renames.map { $0.colName } == ["a", "b", "a"])
+    #expect(renames.map { $0.newColName } == ["b", "c", "d"])
+  }
 }
diff --git a/Tests/SparkConnectTests/DataFrameTests.swift b/Tests/SparkConnectTests/DataFrameTests.swift
index 063a359..8687a1b 100644
--- a/Tests/SparkConnectTests/DataFrameTests.swift
+++ b/Tests/SparkConnectTests/DataFrameTests.swift
@@ -635,6 +635,16 @@ struct DataFrameTests {
     await spark.stop()
   }
 
+  @Test
+  func withColumnRenamedWithDuplicateNames() async throws {
+    let spark = try await SparkSession.builder.getOrCreate()
+    let df = try await spark.sql("SELECT 1 a, 2 b, 3 c, 4 d")
+    // Renames are applied in the given order. The second `a` is ignored because `a` is renamed.
+    #expect(try await df.withColumnRenamed(["a", "a"], ["x", "y"]).columns == ["x", "b", "c", "d"])
+    #expect(try await df.withColumnRenamed(["a", "x"], ["x", "y"]).columns == ["y", "b", "c", "d"])
+    await spark.stop()
+  }
+
   @Test
   func withColumn() async throws {
     let spark = try await SparkSession.builder.getOrCreate()


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org