You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/09/03 20:54:39 UTC
[PR] [SPARK-59223] Support VARIANT functions [spark-connect-swift]
dongjoon-hyun opened a new pull request, #526:
URL: https://github.com/apache/spark-connect-swift/pull/526
### What changes were proposed in this pull request?
This PR aims to support 16 `VARIANT` functions (24 overloads).
| Function | Since | Signature |
| --- | --- | --- |
| `parse_json` | 4.0.0 | `(Column)` |
| `try_parse_json` | 4.0.0 | `(Column)` |
| `to_variant_object` | 4.0.0 | `(Column)` |
| `is_variant_null` | 4.0.0 | `(Column)` |
| `variant_get` | 4.0.0 | `(Column, String \| Column, String)` |
| `try_variant_get` | 4.0.0 | `(Column, String \| Column, String)` |
| `schema_of_variant` | 4.0.0 | `(Column)` |
| `schema_of_variant_agg` | 4.0.0 | `(Column)` |
| `is_valid_variant` | 4.2.0 | `(Column)` |
| `variant_insert` | 4.3.0 | `(Column, String \| Column, Column)` |
| `try_variant_insert` | 4.3.0 | `(Column, String \| Column, Column)` |
| `variant_set` | 4.3.0 | `(Column, String \| Column, Column, Bool = true)` |
| `try_variant_set` | 4.3.0 | `(Column, String \| Column, Column, Bool = true)` |
| `variant_array_append` | 4.3.0 | `(Column, String \| Column, Column)` |
| `try_variant_array_append` | 4.3.0 | `(Column, String \| Column, Column)` |
| `variant_strip_nulls` | 4.3.0 | `(Column, Bool = true)` |
The functions live in a new `VariantFunctions.swift`, except `schema_of_variant_agg`
which is an aggregate function and is placed in `AggregateFunctions.swift`.
Where the upstream `path` argument is `Union[Column, str]`, both a `String` and a `Column`
overload are provided, following the existing `schema_of_json` precedent. Like PySpark's
Spark Connect client, the `createIfMissing` and `includeArrays` flags are always sent as
literal arguments.
`variant_delete` is out of scope because it is introduced in Spark 5.0.0.
### Why are the changes needed?
To improve API coverage. `DataType` already models `.variant`, but no `VARIANT` function
was available in this client.
### Does this PR introduce _any_ user-facing change?
No, this is a new addition to the API.
### How was this patch tested?
Pass the CIs with the newly added test suite, `VariantFunctionsTests`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59223] Support VARIANT functions [spark-connect-swift]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #526:
URL: https://github.com/apache/spark-connect-swift/pull/526#issuecomment-5532400746
Merged to main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59223] Support VARIANT functions [spark-connect-swift]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #526: [SPARK-59223] Support VARIANT functions
URL: https://github.com/apache/spark-connect-swift/pull/526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59223] Support VARIANT functions [spark-connect-swift]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #526:
URL: https://github.com/apache/spark-connect-swift/pull/526#issuecomment-5532544167
> Thank you and sorry for too many PRs, @viirya .
No worries. Thanks for working on these PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59223] Support VARIANT functions [spark-connect-swift]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #526:
URL: https://github.com/apache/spark-connect-swift/pull/526#issuecomment-5532395731
Thank you and sorry for too many PRs, @viirya .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org