[PR] [SPARK-59216] Support JSON and CSV functions [spark-connect-swift]

["dongjoon-hyun (via GitHub)" <gi...@apache.org>]

dongjoon-hyun opened a new pull request, #522:
URL: https://github.com/apache/spark-connect-swift/pull/522

   ### What changes were proposed in this pull request?
   
   This PR aims to support the following 10 JSON and CSV functions in two new files,
   `JsonFunctions.swift` and `CsvFunctions.swift`.
   
   **JSON (7)**: `from_json`, `to_json`, `schema_of_json`, `get_json_object`, `json_tuple`,
   `json_array_length`, `json_object_keys`
   
   **CSV (3)**: `from_csv`, `to_csv`, `schema_of_csv`
   
   Unlike the previous function batches, these functions take an option map. Apache Spark
   encodes it as a plain trailing argument rather than a dedicated proto field: both
   `Column.fnWithOptions` in `sql/api/src/main/scala/org/apache/spark/sql/Column.scala` and
   `_options_to_col` in `python/pyspark/sql/connect/functions/builtin.py` flatten the map into
   `k1, v1, k2, v2, ...`, wrap it in a `map(...)` function call, and append it as the last
   argument. When there is no option, no extra argument is appended at all. This PR mirrors
   that exactly with a new internal `fn(_ name:options:_ args:)` helper in `Functions.swift`,
   sorting the keys so that the generated expression is deterministic (Swift `Dictionary` is
   unordered).
   
   For the `schema` argument, this PR follows the existing
   `DataFrameReader.schema(_ schema: String)` / `schema(_ schema: StructType)` precedent:
   `from_json` accepts a DDL `String`, a `StructType` (converted via `StructType.toDDL`), or a
   `Column` such as a `schema_of_json` result. `from_csv` accepts a DDL `String` or a `Column`,
   matching the Spark Connect Python client. `ArrayType` and `MapType` schemas are covered by
   the DDL string form, e.g. `ARRAY<STRUCT<a: INT>>`.
   
   `json_tuple` is a generator function, and is implemented like the existing `explode` /
   `posexplode` / `inline` functions.
   
   ### Why are the changes needed?
   
   To improve the API coverage of the Swift Spark Connect client. Currently, no JSON or CSV
   function is supported, so Swift users cannot parse or produce JSON and CSV strings inside a
   `Column` expression without falling back to `selectExpr`.
   
   All the added functions were introduced in Apache Spark 3.5.0 or earlier, so no server
   version gate is needed.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No, this is a new feature which adds 10 new functions.
   
   ```swift
   let df = try await spark.sql("SELECT 1 AS id, 'a' AS name")
   let parsed = from_json(to_json(struct(col("id"), col("name"))), "id INT, name STRING")
   try await df.select(parsed.alias("parsed")).selectExpr("parsed.id", "parsed.name").show()
   
   try await spark.range(1).select(schema_of_json("{\"a\": 1, \"b\": \"x\"}")).show()
   // STRUCT<a: BIGINT, b: STRING>
   
   try await df.select(to_csv(struct(col("id"), col("name")), ["sep": ";"])).show()
   // 1;a
   ```
   
   ### How was this patch tested?
   
   Pass the CIs with the newly added test suites, `JsonFunctionsTests` and `CsvFunctionsTests`.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Opus 5


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-59216] Support JSON and CSV functions [spark-connect-swift]

["dongjoon-hyun (via GitHub)" <gi...@apache.org>]

dongjoon-hyun closed pull request #522: [SPARK-59216] Support JSON and CSV functions
URL: https://github.com/apache/spark-connect-swift/pull/522


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-59216] Support JSON and CSV functions [spark-connect-swift]

["dongjoon-hyun (via GitHub)" <gi...@apache.org>]

dongjoon-hyun commented on PR #522:
URL: https://github.com/apache/spark-connect-swift/pull/522#issuecomment-5531149458

   Merged to main


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-59216] Support JSON and CSV functions [spark-connect-swift]

["dongjoon-hyun (via GitHub)" <gi...@apache.org>]

dongjoon-hyun commented on PR #522:
URL: https://github.com/apache/spark-connect-swift/pull/522#issuecomment-5531087955

   Thank you, @viirya ! 😄 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org