You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "marcuslin123 (via GitHub)" <gi...@apache.org> on 2026/07/16 21:43:32 UTC
[PR] [SPARK-58062][SQL] Add variant_strip_nulls expression [spark]
marcuslin123 opened a new pull request, #57318:
URL: https://github.com/apache/spark/pull/57318
### What changes were proposed in this pull request?
Add a new built-in SQL function `variant_strip_nulls(v) -> variant` that returns a variant value with all variant-null values removed from its objects, applied recursively. Returns NULL if the input is NULL.
```sql
SELECT variant_strip_nulls(parse_json('{"a":1, "b":null}')); -- {"a":1}
SELECT variant_strip_nulls(parse_json('{"a":{"b":null, "c":2}}')); -- {"a":{"c":2}}
SELECT variant_strip_nulls(parse_json('[1, null, 2]')); -- [1,null,2]
```
The function is exposed across all surfaces: SQL, the Scala DataFrame API (`variant_strip_nulls(col)`), and PySpark (`F.variant_strip_nulls(col)`), including Spark Connect.
**Implementation:** Mirrors the existing `variant_delete` (`VariantDelete` / `VariantBuilder`) pattern. The core logic is `VariantBuilder.stripNulls`, which recursively rebuilds the variant into a fresh `VariantBuilder`, dropping object fields whose value is a variant null. The expression is a `RuntimeReplaceable` that delegates to `VariantExpressionEvalUtils.variantStripNulls`, so interpreted eval and codegen both come for free.
### Design decisions (open to reviewer feedback)
The JIRA did not specify exact semantics, so I chose what I believe is the least-surprising behavior:
1. **Recursive.** The purpose of `variant_strip_nulls` is to remove null-valued fields from the variant, so it is applied at every level of nesting rather than only the top level. Stripping only the top level would inconsistently leave nested nulls behind.
2. **Array elements are preserved.** Array positions are meaningful — removing a null element would shift the indices of every subsequent element and silently change the data. So the function recurses *into* array elements (to strip nulls from nested objects) but never removes the elements themselves. A variant null that is an array element is therefore kept.
3. **Only object-field variant nulls are stripped.** A SQL `NULL` input returns `NULL` (handled by the expression layer).
Happy to adjust any of these if reviewers prefer different semantics.
### Does this PR introduce _any_ user-facing change?
Yes. A new `variant_strip_nulls` function is available in SQL, the Scala DataFrame API, and PySpark (classic and Connect).
### How was this patch tested?
- Catalyst unit test in `VariantExpressionSuite` covering: no-op cases, top-level null stripping, recursive nested-object stripping, array elements preserved, and nested objects inside arrays stripped.
- End-to-end test in `VariantEndToEndSuite` exercising the SQL string form, the Scala function, and codegen.
- Spark Connect plan golden files regenerated.
- Expression schema golden file regenerated.
- PySpark doctest.
### Was this patch authored or co-authored using generative AI tooling?
Generative AI tooling (Claude Code) was used as an assistive tool for implementation guidance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58062][SQL] Add variant_strip_nulls expression [spark]
Posted by "bojana-db (via GitHub)" <gi...@apache.org>.
bojana-db commented on PR #57318:
URL: https://github.com/apache/spark/pull/57318#issuecomment-5025317120
> @bojana-db please help coordinate this effort - do you intend to proceed with #56864?
Yes, `variant_strip_nulls` is done as part of the variant manipulation functions effort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58062][SQL] Add variant_strip_nulls expression [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57318:
URL: https://github.com/apache/spark/pull/57318#discussion_r3611495808
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:
##########
@@ -119,6 +119,42 @@ case class IsVariantNull(child: Expression) extends UnaryExpression
copy(child = newChild)
}
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(v) - Returns a variant value with all the null-valued fields removed from its variant objects, applied recursively. Returns NULL if the input is NULL.",
Review Comment:
Let's problaby add the descriptions for arguments as well. I am trying to document them all. Please refer to https://github.com/apache/spark/pull/57185 https://github.com/apache/spark/pull/57184
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58062][SQL] Add variant_strip_nulls expression [spark]
Posted by "marcuslin123 (via GitHub)" <gi...@apache.org>.
marcuslin123 closed pull request #57318: [SPARK-58062][SQL] Add variant_strip_nulls expression
URL: https://github.com/apache/spark/pull/57318
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58062][SQL] Add variant_strip_nulls expression [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57318:
URL: https://github.com/apache/spark/pull/57318#issuecomment-5021829594
@bojana-db please help coordinate this effort - do you intend to proceed with https://github.com/apache/spark/pull/56864?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58062][SQL] Add variant_strip_nulls expression [spark]
Posted by "marcuslin123 (via GitHub)" <gi...@apache.org>.
marcuslin123 commented on PR #57318:
URL: https://github.com/apache/spark/pull/57318#issuecomment-5060490884
Closing as a duplicate of #56864, which predates this PR and is part of the broader variant manipulation functions effort. Thanks @bojana-db @uros-b @HyukjinKwon for the coordination.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58062][SQL] Add variant_strip_nulls expression [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57318:
URL: https://github.com/apache/spark/pull/57318#discussion_r3611495808
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:
##########
@@ -119,6 +119,42 @@ case class IsVariantNull(child: Expression) extends UnaryExpression
copy(child = newChild)
}
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(v) - Returns a variant value with all the null-valued fields removed from its variant objects, applied recursively. Returns NULL if the input is NULL.",
Review Comment:
Let's problaby add the descriptions for arguments as well. I am trying to document them all. Please refer to https://github.com/apache/spark/pull/57185 https://github.com/apache/spark/pull/57184 https://github.com/apache/spark/pull/57183
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org