You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/09/11 23:07:26 UTC
[PR] [SPARK-59450] Support Codable in `createDataFrame` and `DataFrame.collect` [spark-connect-swift]
dongjoon-hyun opened a new pull request, #552:
URL: https://github.com/apache/spark-connect-swift/pull/552
### What changes were proposed in this pull request?
This PR aims to support `Codable`-based DataFrame creation and collection in `SparkSession` and `DataFrame`, leveraging the in-tree `ArrowEncoder` and `ArrowDecoder`.
Specifically:
1. **`SparkSession.createDataFrame` with `Encodable`**:
- `createDataFrame<T: Encodable>(_ data: [T]) async throws -> DataFrame`: Automatically encodes an array of `Encodable` instances into an Apache Arrow `RecordBatch` via `ArrowEncoder`, infers the Spark DDL schema from the batch schema, and builds a `LocalRelation` or `CachedLocalRelation` (for payloads >= 1MiB).
- `createDataFrame<T: Encodable>(_ data: [T], _ schema: String)` and `createDataFrame<T: Encodable>(_ data: [T], _ schema: StructType)`: Supports explicit schema overrides and empty datasets with schema.
2. **`DataFrame.collect(as:)` with `Decodable`**:
- `collect<T: Decodable>(as type: T.Type = T.self) async throws -> [T]`: Executes the plan and directly decodes each Arrow `RecordBatch` into `[T]` using `ArrowDecoder`, eliminating intermediate untyped `Row` allocations and dictionary boxing.
3. **`ArrowEncoder` and `ArrowDecoder` enhancements**:
- Added support for Swift native `Int` (mapped to `Int64`) and `UInt` (mapped to `UInt64`) across keyed, unkeyed, and single-value containers.
- Added support for decoding `Date` and `TimestampNanos` from Arrow `Timestamp` columns (seconds, milliseconds, microseconds, and nanoseconds).
4. **Schema mapping utilities**:
- Added `DataType.init(_ arrowType: ArrowType) throws` and `StructType.init(_ arrowSchema: ArrowSchema) throws` to translate Arrow schemas to Spark SQL data types.
```swift
struct Person: Codable, Sendable, Equatable {
let name: String
let age: Int
}
let people = [Person(name: "Alice", age: 20), Person(name: "Bob", age: 25)]
// 1. Create DataFrame directly from Swift models (auto-inferred schema)
let df = try await spark.createDataFrame(people)
// 2. Collect query results directly into Swift models
let results: [Person] = try await df.filter("age >= 21").collect(as: Person.self)
```
### Why are the changes needed?
Currently, `SparkSession.createDataFrame` only accepts untyped rows `[[Sendable?]]` with an explicit DDL string or `StructType` schema, and `DataFrame.collect()` only returns untyped `[Row]`.
These new APIs provide a type-safe, Dataset-like programming experience in Swift, allowing developers to work seamlessly with native Swift `Codable` structs.
### Does this PR introduce _any_ user-facing change?
No behavior change because this PR adds only new public generic methods.
### How was this patch tested?
Pass the CIs with the newly added test cases.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Gemini 3.8 Flash (High)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59450] Support Codable in `createDataFrame` and `DataFrame.collect` [spark-connect-swift]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #552:
URL: https://github.com/apache/spark-connect-swift/pull/552#issuecomment-5642599938
Thank you, @viirya !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59450] Support Codable in `createDataFrame` and `DataFrame.collect` [spark-connect-swift]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #552: [SPARK-59450] Support Codable in `createDataFrame` and `DataFrame.collect`
URL: https://github.com/apache/spark-connect-swift/pull/552
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59450] Support Codable in `createDataFrame` and `DataFrame.collect` [spark-connect-swift]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #552:
URL: https://github.com/apache/spark-connect-swift/pull/552#issuecomment-5643369638
Merged to main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org