(spark-connect-swift) branch main updated: [SPARK-59484] Use `value` as the output column name of `DataFrame.toJSON`

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git


The following commit(s) were added to refs/heads/main by this push:
     new c0bf10a  [SPARK-59484] Use `value` as the output column name of `DataFrame.toJSON`
c0bf10a is described below

commit c0bf10a1480f6a6d5a94ed9ea1e565a9ef7118cc
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Sun Sep 13 23:38:30 2026 -0700

    [SPARK-59484] Use `value` as the output column name of `DataFrame.toJSON`
    
    ### What changes were proposed in this pull request?
    
    This PR aims to make `DataFrame.toJSON()` return a single column named `value`, like the Scala Spark Connect client.
    
    ```swift
    -    return selectExpr("to_json(struct(*))")
    +    return selectExpr("to_json(struct(*)) AS value")
    ```
    
    ### Why are the changes needed?
    
    The Scala Spark Connect client names the `toJSON` output column `value`.
    
    - [Apache Spark 4.2.0 (2026-07-11) `Dataset.toJSON`](https://github.com/apache/spark/blob/v4.2.0/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala#L1277-L1279)
    
    ```scala
    def toJSON: Dataset[String] = {
      select(to_json(struct(col("*"))).as("value")).as(StringEncoder)
    }
    ```
    
    The Swift client currently names the column after the expression, such as `to_json(struct(id))`. So the column name depends on the input schema, and code ported from Scala that refers to `value` fails.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The output column is renamed from `to_json(struct(...))` to `value`, matching the Scala Spark Connect client. The JSON content is unchanged.
    
    **BEFORE**
    
    ```swift
    try await spark.range(2).toJSON().columns  // ["to_json(struct(id))"]
    ```
    
    **AFTER**
    
    ```swift
    try await spark.range(2).toJSON().columns  // ["value"]
    try await spark.range(2).toJSON().schema   // value STRING (nullable)
    ```
    
    ### How was this patch tested?
    
    Pass the CIs with the updated test case. It checks the column name, the schema (`value STRING`, nullable), unchanged JSON content, a multi-column DataFrame, and a nested struct column.
    
    I also tested manually with an Apache Spark 4.2.0 Spark Connect server.
    
    ```
    $ swift test --no-parallel --filter DataFrameTests
    ...
    Test run with 152 tests in 2 suites passed after 22.965 seconds.
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5
    
    Closes #565 from dongjoon-hyun/SPARK-59484.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 Sources/SparkConnect/DataFrame+Transformations.swift |  4 ++--
 Tests/SparkConnectTests/DataFrameTests.swift         | 13 ++++++++++++-
 2 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/Sources/SparkConnect/DataFrame+Transformations.swift b/Sources/SparkConnect/DataFrame+Transformations.swift
index 80ebd5d..e40e25f 100644
--- a/Sources/SparkConnect/DataFrame+Transformations.swift
+++ b/Sources/SparkConnect/DataFrame+Transformations.swift
@@ -155,9 +155,9 @@ extension DataFrame {
   }
 
   /// Returns the content of the Dataset as a Dataset of JSON strings.
-  /// - Returns: A ``DataFrame`` with a single string column whose content is JSON.
+  /// - Returns: A ``DataFrame`` with a single string column named `value` whose content is JSON.
   public func toJSON() -> DataFrame {
-    return selectExpr("to_json(struct(*))")
+    return selectExpr("to_json(struct(*)) AS value")
   }
 
   /// Projects a set of expressions and returns a new ``DataFrame``.
diff --git a/Tests/SparkConnectTests/DataFrameTests.swift b/Tests/SparkConnectTests/DataFrameTests.swift
index 4d2c9d2..332a6a9 100644
--- a/Tests/SparkConnectTests/DataFrameTests.swift
+++ b/Tests/SparkConnectTests/DataFrameTests.swift
@@ -1565,11 +1565,22 @@ struct DataFrameTests {
   func toJSON() async throws {
     let spark = try await SparkSession.builder.getOrCreate()
     let df = try await spark.range(2).toJSON()
-    #expect(try await df.columns == ["to_json(struct(id))"])
+    #expect(try await df.columns == ["value"])
+    #expect(
+      try await df.schema
+        == StructType(fields: [StructField(name: "value", dataType: .string, nullable: true)]))
     #expect(try await df.collect() == [Row("{\"id\":0}"), Row("{\"id\":1}")])
 
     let expected = [Row("{\"a\":1,\"b\":2,\"c\":3}")]
     #expect(try await spark.sql("SELECT 1 a, 2 b, 3 c").toJSON().collect() == expected)
+
+    let df2 = try await spark.sql("SELECT 'Alice' AS name, 20 AS age").toJSON()
+    #expect(try await df2.columns == ["value"])
+    #expect(try await df2.collect() == [Row("{\"name\":\"Alice\",\"age\":20}")])
+
+    let df3 = try await spark.sql("SELECT 1 AS id, named_struct('a', 1, 'b', 'x') AS s").toJSON()
+    #expect(try await df3.columns == ["value"])
+    #expect(try await df3.collect() == [Row("{\"id\":1,\"s\":{\"a\":1,\"b\":\"x\"}}")])
     await spark.stop()
   }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org