You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "peterxcli (via GitHub)" <gi...@apache.org> on 2026/08/23 05:32:20 UTC
[I] Support native is_variant_null and Spark 4.2 is_valid_variant [datafusion-comet]
peterxcli opened a new issue, #5429:
URL: https://github.com/apache/datafusion-comet/issues/5429
### What is the problem the feature request solves?
Comet has no native predicate for distinguishing Variant JSON null from SQL NULL. Spark 4.0+ provides `is_variant_null`, and Spark 4.2 adds `is_valid_variant` for validating the Variant value/metadata pair.
```sql
SELECT is_variant_null(v) FROM t;
SELECT is_valid_variant(v) FROM t; -- Spark 4.2+
```
Spark's [`is_variant_null`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L80-L119) returns true only for Variant null and false for SQL NULL and all other values; malformed physical data raises an error in its [evaluation helper](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala#L58-L70). Spark 4.2's [`is_valid_variant`](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L957-L987) returns true/false for a non-null Variant and SQL NULL for SQL NULL.
### Describe the potential solution
Add expression-specific serializers and native evaluators over the canonical `[value, metadata]` representation:
- implement Spark's exact JSON-null, SQL-NULL, malformed-value, and nullability behavior;
- validate both Variant value and metadata bytes for `is_valid_variant`;
- expose `is_valid_variant` only in the Spark 4.2 shim/profile while leaving Spark 4.0/4.1 compilation and registry behavior unchanged; and
- keep Variant rejected for unrelated predicates/operators.
Add focused parity and native-plan tests for every Variant scalar/container kind, Variant null, SQL NULL, malformed value bytes, malformed metadata bytes, and the Spark 4.2 version boundary.
### Additional context
Spark 4.2 registers `is_valid_variant` alongside the existing Variant functions in its [tagged function registry](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L973-L983). It is not present in Spark 4.0/4.1.
Related work: #4295, #5407, #5424, and #5425.
Non-goals: comparisons, hash functions, parsing, casts, Variant mutation functions, C2R, shuffle/spill, writing, and Python transport.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support native is_variant_null and Spark 4.2 is_valid_variant [datafusion-comet]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on issue #5429:
URL: https://github.com/apache/datafusion-comet/issues/5429#issuecomment-5384432592
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org