You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2026/08/11 02:40:53 UTC
(spark) branch master updated: [SPARK-51975][SQL] Add variant_from_arrays and variant_from_entries
This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5d23405f88a8 [SPARK-51975][SQL] Add variant_from_arrays and variant_from_entries
5d23405f88a8 is described below
commit 5d23405f88a85ee890260f90256e8f0ef8a77225
Author: SreeramaYeshwanthGowd <ye...@gmail.com>
AuthorDate: Tue Aug 11 10:40:25 2026 +0800
[SPARK-51975][SQL] Add variant_from_arrays and variant_from_entries
### What changes were proposed in this pull request?
Add two built in scalar SQL functions that construct a Variant object directly from collections:
- `variant_from_arrays(keys, values)` builds a Variant object from an array of string keys and an array of values.
- `variant_from_entries(entries)` builds a Variant object from an array of key/value struct entries (a two field struct per entry).
```
> SELECT variant_from_arrays(array('a', 'b'), array(1, 2));
{"a":1,"b":2}
> SELECT variant_from_entries(array(struct('a', 1), struct('b', 2)));
{"a":1,"b":2}
```
API surface added:
- SQL: `variant_from_arrays`, `variant_from_entries`
- Scala DataFrame: `functions.variant_from_arrays(keys, values)`, `functions.variant_from_entries(entries)`
- PySpark, classic and Spark Connect: `pyspark.sql.functions.variant_from_arrays` / `variant_from_entries`
Implementation notes:
- Both are native expressions in `variantExpressions.scala` next to `to_variant_object`, delegating (eval and codegen) to two new `VariantExpressionEvalUtils` helpers that build the object in a single pass with `VariantBuilder` (`addKey` / `finishWritingObject`) and reuse the existing `buildVariant` for each value. This avoids materializing an intermediate `MapType`, which is the point of the ticket.
- Semantics, chosen to match the existing `to_variant_object` and `map_from_*` precedents:
- Object keys must be non-null strings. A non-string key type is rejected at analysis (as `to_variant_object` does for non-string-keyed maps, it is not implicitly cast); a null key raises `NULL_MAP_KEY`.
- Duplicate keys raise `VARIANT_DUPLICATE_KEY`, matching `to_variant_object` (Variant objects cannot hold duplicate keys). This is stricter than `map_from_*` under the non default `spark.sql.mapKeyDedupPolicy=LAST_WIN`.
- Null values are kept as Variant null. A null array argument yields `NULL`, and for `variant_from_entries` a null entry yields `NULL` (as `map_from_entries` does).
- For `variant_from_arrays`, a keys/values length mismatch reuses the same error `map_from_arrays` raises.
### Why are the changes needed?
Today, creating a Variant object from arrays or entries requires a two step expression: `to_variant_object(map_from_arrays(...))` or `to_variant_object(map_from_entries(...))`. That materializes an intermediate `MapType` only to immediately convert it. `variant_from_arrays` / `variant_from_entries` build the Variant object directly, reducing allocation and CPU, which matters for large maps and high volume workloads. They also mirror Spark's own `map_from_arrays` / `map_from_entries` n [...]
### Does this PR introduce _any_ user-facing change?
Yes. It adds two new built in SQL functions and their Scala and PySpark DataFrame API entries. No existing behavior changes.
### How was this patch tested?
- Catalyst unit tests in `VariantExpressionSuite` covering object construction, key sorting, empty input, null values, nested values, null array / null entry inputs returning null, and the null key / duplicate key / length mismatch errors.
- An end to end test in `VariantEndToEndSuite` exercising the DataFrame API under both `CODEGEN_ONLY` and `NO_CODEGEN` over a non foldable relation.
- SQL golden tests in `variant/variant-from-arrays-entries.sql`, covering each JSON type, null handling, and the null key / duplicate key / length mismatch / non-string key / unsupported value type errors.
- PySpark doctests and a `test_functions.py` test, and regenerated `sql-expression-schema.md`.
### Was this patch authored or co-authored using generative AI tooling? No
Closes #57497 from SreeramaYeshwanthGowd/add-variant-from-arrays-entries.
Authored-by: SreeramaYeshwanthGowd <ye...@gmail.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
.../source/reference/pyspark.sql/functions.rst | 2 +
python/pyspark/sql/connect/functions/builtin.py | 14 +
python/pyspark/sql/functions/__init__.py | 2 +
python/pyspark/sql/functions/builtin.py | 69 +++++
python/pyspark/sql/tests/test_functions.py | 17 ++
.../scala/org/apache/spark/sql/functions.scala | 30 +++
.../sql/catalyst/analysis/FunctionRegistry.scala | 2 +
.../variant/VariantExpressionEvalUtils.scala | 71 +++++
.../expressions/variant/variantExpressions.scala | 155 ++++++++++-
.../variant/VariantExpressionSuite.scala | 64 +++++
.../apache/spark/sql/PlanGenerationTestSuite.scala | 8 +
.../function_variant_from_arrays.explain | 2 +
.../function_variant_from_entries.explain | 2 +
.../queries/function_variant_from_arrays.json | 169 ++++++++++++
.../queries/function_variant_from_arrays.proto.bin | Bin 0 -> 1504 bytes
.../queries/function_variant_from_entries.json | 192 +++++++++++++
.../function_variant_from_entries.proto.bin | Bin 0 -> 1704 bytes
.../sql-functions/sql-expression-schema.md | 2 +
.../variant/variant-from-arrays-entries.sql.out | 244 +++++++++++++++++
.../inputs/variant/variant-from-arrays-entries.sql | 47 ++++
.../variant/variant-from-arrays-entries.sql.out | 296 +++++++++++++++++++++
.../apache/spark/sql/VariantEndToEndSuite.scala | 25 ++
.../collation/CollationExpressionWalkerSuite.scala | 6 +-
23 files changed, 1417 insertions(+), 2 deletions(-)
diff --git a/python/docs/source/reference/pyspark.sql/functions.rst b/python/docs/source/reference/pyspark.sql/functions.rst
index f3d973b1e735..40b901998406 100644
--- a/python/docs/source/reference/pyspark.sql/functions.rst
+++ b/python/docs/source/reference/pyspark.sql/functions.rst
@@ -611,6 +611,8 @@ VARIANT Functions
variant_array_append
try_variant_array_append
variant_delete
+ variant_from_arrays
+ variant_from_entries
variant_get
variant_insert
try_variant_insert
diff --git a/python/pyspark/sql/connect/functions/builtin.py b/python/pyspark/sql/connect/functions/builtin.py
index b2f1149db00d..3a6fe6c93d23 100644
--- a/python/pyspark/sql/connect/functions/builtin.py
+++ b/python/pyspark/sql/connect/functions/builtin.py
@@ -2211,6 +2211,20 @@ def to_variant_object(col: "ColumnOrName") -> Column:
to_variant_object.__doc__ = pysparkfuncs.to_variant_object.__doc__
+def variant_from_arrays(keys: "ColumnOrName", values: "ColumnOrName") -> Column:
+ return _invoke_function_over_columns("variant_from_arrays", keys, values)
+
+
+variant_from_arrays.__doc__ = pysparkfuncs.variant_from_arrays.__doc__
+
+
+def variant_from_entries(entries: "ColumnOrName") -> Column:
+ return _invoke_function_over_columns("variant_from_entries", entries)
+
+
+variant_from_entries.__doc__ = pysparkfuncs.variant_from_entries.__doc__
+
+
def parse_json(col: "ColumnOrName") -> Column:
return _invoke_function("parse_json", _to_col(col))
diff --git a/python/pyspark/sql/functions/__init__.py b/python/pyspark/sql/functions/__init__.py
index 696691884f69..ad722c3fcc22 100644
--- a/python/pyspark/sql/functions/__init__.py
+++ b/python/pyspark/sql/functions/__init__.py
@@ -490,6 +490,8 @@ __all__ = [ # noqa: F405
"variant_array_append",
"try_variant_array_append",
"variant_delete",
+ "variant_from_arrays",
+ "variant_from_entries",
"variant_get",
"variant_insert",
"try_variant_insert",
diff --git a/python/pyspark/sql/functions/builtin.py b/python/pyspark/sql/functions/builtin.py
index 30d07989b234..07248f6b5650 100644
--- a/python/pyspark/sql/functions/builtin.py
+++ b/python/pyspark/sql/functions/builtin.py
@@ -22777,6 +22777,75 @@ def to_variant_object(
return _invoke_function("to_variant_object", _to_java_column(col))
+@_try_remote_functions
+def variant_from_arrays(keys: "ColumnOrName", values: "ColumnOrName") -> Column:
+ """
+ Creates a variant object from the given arrays of keys and values. The keys must be non-null
+ strings and the two arrays must have the same length.
+
+ .. versionadded:: 4.4.0
+
+ Parameters
+ ----------
+ keys : :class:`~pyspark.sql.Column` or column name
+ an array of string keys.
+ values : :class:`~pyspark.sql.Column` or column name
+ an array of values.
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ a new column of VariantType.
+
+ See Also
+ --------
+ :meth:`pyspark.sql.functions.variant_from_entries`
+ :meth:`pyspark.sql.functions.to_variant_object`
+
+ Examples
+ --------
+ >>> from pyspark.sql import functions as sf
+ >>> df = spark.sql("SELECT array('a', 'b') AS keys, array(1, 2) AS values")
+ >>> df.select(sf.variant_from_arrays("keys", "values").cast("string").alias("r")).collect()
+ [Row(r='{"a":1,"b":2}')]
+ """
+ return _invoke_function_over_columns("variant_from_arrays", keys, values)
+
+
+@_try_remote_functions
+def variant_from_entries(entries: "ColumnOrName") -> Column:
+ """
+ Creates a variant object from an array of key/value struct entries. The keys must be non-null
+ strings.
+
+ .. versionadded:: 4.4.0
+
+ Parameters
+ ----------
+ entries : :class:`~pyspark.sql.Column` or column name
+ an array of key/value structs, where the first field is a string key and the second field
+ is the value.
+
+ Returns
+ -------
+ :class:`~pyspark.sql.Column`
+ a new column of VariantType.
+
+ See Also
+ --------
+ :meth:`pyspark.sql.functions.variant_from_arrays`
+ :meth:`pyspark.sql.functions.to_variant_object`
+
+ Examples
+ --------
+ >>> from pyspark.sql import functions as sf
+ >>> df = spark.sql("SELECT array(struct('a', 1), struct('b', 2)) AS entries")
+ >>> df.select(sf.variant_from_entries("entries").cast("string").alias("r")).collect()
+ [Row(r='{"a":1,"b":2}')]
+ """
+ return _invoke_function_over_columns("variant_from_entries", entries)
+
+
@_try_remote_functions
def parse_json(
col: "ColumnOrName",
diff --git a/python/pyspark/sql/tests/test_functions.py b/python/pyspark/sql/tests/test_functions.py
index 41764083c3c8..3bdfec72eb04 100644
--- a/python/pyspark/sql/tests/test_functions.py
+++ b/python/pyspark/sql/tests/test_functions.py
@@ -3724,6 +3724,23 @@ class FunctionsTestsMixin:
).collect()
self.assertEqual("""{"a":1}""", actual[0]["var"])
+ def test_variant_from_arrays_and_entries(self):
+ df = self.spark.createDataFrame(
+ [(["a", "b"], [1, 2])], "keys array<string>, values array<int>"
+ )
+ actual = df.select(
+ F.to_json(F.variant_from_arrays("keys", "values")).alias("var"),
+ ).collect()
+ self.assertEqual("""{"a":1,"b":2}""", actual[0]["var"])
+
+ df2 = self.spark.createDataFrame(
+ [([("a", 1), ("b", 2)],)], "entries array<struct<k string, v int>>"
+ )
+ actual2 = df2.select(
+ F.to_json(F.variant_from_entries("entries")).alias("var"),
+ ).collect()
+ self.assertEqual("""{"a":1,"b":2}""", actual2[0]["var"])
+
def test_schema_of_csv(self):
with self.assertRaises(PySparkTypeError) as pe:
F.schema_of_csv(1)
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/functions.scala b/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
index 9e6265aabd68..1c0b3e4ccbd9 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/functions.scala
@@ -14222,6 +14222,36 @@ object functions {
*/
def to_variant_object(col: Column): Column = Column.fn("to_variant_object", col)
+ /**
+ * Creates a variant object from the given arrays of keys and values. The keys must be non-null
+ * strings and the two arrays must have the same length.
+ *
+ * @param keys
+ * a column that evaluates to an array of string keys.
+ * @param values
+ * a column that evaluates to an array of values.
+ * @group variant_funcs
+ * @since 4.4.0
+ * @return
+ * Returns a column that evaluates to a variant.
+ */
+ def variant_from_arrays(keys: Column, values: Column): Column =
+ Column.fn("variant_from_arrays", keys, values)
+
+ /**
+ * Creates a variant object from an array of key/value struct entries. The keys must be non-null
+ * strings.
+ *
+ * @param entries
+ * a column that evaluates to an array of key/value structs.
+ * @group variant_funcs
+ * @since 4.4.0
+ * @return
+ * Returns a column that evaluates to a variant.
+ */
+ def variant_from_entries(entries: Column): Column =
+ Column.fn("variant_from_entries", entries)
+
/**
* Check if a variant value is a variant null. Returns true if and only if the input is a
* variant null and false otherwise (including in the case of SQL NULL).
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
index 7950f90016ad..ef55fb8baee2 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
@@ -992,6 +992,8 @@ object FunctionRegistry {
expression[SchemaOfVariant]("schema_of_variant"),
expression[SchemaOfVariantAgg]("schema_of_variant_agg"),
expression[ToVariantObject]("to_variant_object"),
+ expression[VariantFromArrays]("variant_from_arrays"),
+ expression[VariantFromEntries]("variant_from_entries"),
expression[IsValidVariant]("is_valid_variant"),
expression[VariantDelete]("variant_delete"),
expressionBuilder("variant_insert", VariantInsertExpressionBuilder),
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala
index da1bfbcea829..d93bf0f62848 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala
@@ -280,6 +280,77 @@ object VariantExpressionEvalUtils {
new VariantVal(v.getValue, v.getMetadata)
}
+ /**
+ * Build a variant object directly from a keys array and a values array, without materializing an
+ * intermediate map. Keys must be non-null strings and the two arrays must have equal length. A
+ * null key raises `NULL_MAP_KEY`, a duplicate key raises `VARIANT_DUPLICATE_KEY` (matching
+ * to_variant_object), and null values are kept as variant null.
+ */
+ def variantFromArrays(keys: ArrayData, values: ArrayData, valueType: DataType): VariantVal = {
+ if (keys.numElements() != values.numElements()) {
+ // Reuse the same error map_from_arrays raises for a keys/values length mismatch.
+ throw QueryExecutionErrors.mapDataKeyArrayLengthDiffersFromValueArrayLengthError()
+ }
+ val builder = new VariantBuilder(false)
+ val start = builder.getWritePos
+ val numElements = keys.numElements()
+ val fields = new java.util.ArrayList[VariantBuilder.FieldEntry](numElements)
+ var i = 0
+ while (i < numElements) {
+ if (keys.isNullAt(i)) {
+ throw QueryExecutionErrors.nullAsMapKeyNotAllowedError()
+ }
+ val key = keys.getUTF8String(i).toString
+ val id = builder.addKey(key)
+ fields.add(new VariantBuilder.FieldEntry(key, id, builder.getWritePos - start))
+ val value = if (values.isNullAt(i)) null else values.get(i, valueType)
+ buildVariant(builder, value, valueType)
+ i += 1
+ }
+ builder.finishWritingObject(start, fields)
+ val v = builder.result()
+ new VariantVal(v.getValue, v.getMetadata)
+ }
+
+ /**
+ * Build a variant object directly from an array of key/value struct entries, without an
+ * intermediate map. Keys must be non-null strings. A null entry makes the whole result null,
+ * and this is checked for every entry before any value is converted, so a null entry always
+ * dominates a conversion failure in an earlier entry (matching `map_from_entries`). A null key
+ * raises `NULL_MAP_KEY`, a duplicate key raises `VARIANT_DUPLICATE_KEY`, and null values are
+ * kept as variant null.
+ */
+ def variantFromEntries(entries: ArrayData, valueType: DataType): VariantVal = {
+ val numElements = entries.numElements()
+ var i = 0
+ while (i < numElements) {
+ if (entries.isNullAt(i)) {
+ return null
+ }
+ i += 1
+ }
+
+ val builder = new VariantBuilder(false)
+ val start = builder.getWritePos
+ val fields = new java.util.ArrayList[VariantBuilder.FieldEntry](numElements)
+ i = 0
+ while (i < numElements) {
+ val entry = entries.getStruct(i, 2)
+ if (entry.isNullAt(0)) {
+ throw QueryExecutionErrors.nullAsMapKeyNotAllowedError()
+ }
+ val key = entry.getUTF8String(0).toString
+ val id = builder.addKey(key)
+ fields.add(new VariantBuilder.FieldEntry(key, id, builder.getWritePos - start))
+ val value = if (entry.isNullAt(1)) null else entry.get(1, valueType)
+ buildVariant(builder, value, valueType)
+ i += 1
+ }
+ builder.finishWritingObject(start, fields)
+ val v = builder.result()
+ new VariantVal(v.getValue, v.getMetadata)
+ }
+
/** Returns `true` if a data type is or has a child variant type. */
def typeContainsVariant(dt: DataType): Boolean = dt match {
case _: VariantType => true
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala
index 723af5a4fe92..d1198b778957 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala
@@ -34,7 +34,7 @@ import org.apache.spark.sql.catalyst.json.JsonInferSchema
import org.apache.spark.sql.catalyst.plans.logical.{FunctionSignature, InputParameter}
import org.apache.spark.sql.catalyst.trees.TreePattern.{TreePattern, VARIANT_GET}
import org.apache.spark.sql.catalyst.trees.UnaryLike
-import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, GenericArrayData, QuotingUtils}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, ArrayData, GenericArrayData, QuotingUtils}
import org.apache.spark.sql.catalyst.util.DateTimeConstants._
import org.apache.spark.sql.errors.{QueryCompilationErrors, QueryErrorsBase, QueryExecutionErrors}
import org.apache.spark.sql.internal.SQLConf
@@ -194,6 +194,159 @@ case class ToVariantObject(child: Expression)
}
}
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(keys, values) - Creates a variant object from the given arrays of keys and values. The keys must be non-null strings and the two arrays must have the same length.",
+ arguments = """
+ Arguments:
+ * keys - An array of non-null strings used as the object keys.
+ * values - An array of values, with the same length as the keys array.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array('a', 'b'), array(1, 2));
+ {"a":1,"b":2}
+ """,
+ since = "4.4.0",
+ group = "variant_funcs")
+// scalastyle:on line.size.limit
+case class VariantFromArrays(left: Expression, right: Expression)
+ extends BinaryExpression
+ with ExpectsInputTypes
+ with QueryErrorsBase {
+ override def nullIntolerant: Boolean = true
+ override def inputTypes: Seq[AbstractDataType] = Seq(ArrayType, ArrayType)
+ override def dataType: DataType = VariantType
+
+ private lazy val valueType: DataType = right.dataType.asInstanceOf[ArrayType].elementType
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ val defaultCheck = super.checkInputDataTypes()
+ if (defaultCheck.isFailure) {
+ defaultCheck
+ } else {
+ left.dataType.asInstanceOf[ArrayType].elementType match {
+ case _: StringType if VariantGet.checkDataType(valueType, allowStructsAndMaps = true) =>
+ TypeCheckResult.TypeCheckSuccess
+ case _: StringType =>
+ DataTypeMismatch(
+ errorSubClass = "CAST_WITHOUT_SUGGESTION",
+ messageParameters =
+ Map("srcType" -> toSQLType(valueType), "targetType" -> toSQLType(VariantType)))
+ case _ =>
+ DataTypeMismatch(
+ errorSubClass = "UNEXPECTED_INPUT_TYPE",
+ messageParameters = Map(
+ "paramIndex" -> ordinalNumber(0),
+ "requiredType" -> toSQLType(ArrayType(StringType)),
+ "inputSql" -> toSQLExpr(left),
+ "inputType" -> toSQLType(left.dataType)))
+ }
+ }
+ }
+
+ override def prettyName: String = "variant_from_arrays"
+
+ override protected def withNewChildrenInternal(
+ newLeft: Expression, newRight: Expression): VariantFromArrays =
+ copy(left = newLeft, right = newRight)
+
+ override protected def nullSafeEval(keyArray: Any, valueArray: Any): Any =
+ VariantExpressionEvalUtils.variantFromArrays(
+ keyArray.asInstanceOf[ArrayData], valueArray.asInstanceOf[ArrayData], valueType)
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ nullSafeCodeGen(ctx, ev, (keyArray, valueArray) => {
+ val cls = variant.VariantExpressionEvalUtils.getClass.getName.stripSuffix("$")
+ val valueTypeArg = ctx.addReferenceObj("valueType", valueType)
+ s"${ev.value} = $cls.variantFromArrays($keyArray, $valueArray, $valueTypeArg);"
+ })
+ }
+}
+
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+ usage = "_FUNC_(entries) - Creates a variant object from an array of key/value struct entries. The keys must be non-null strings.",
+ arguments = """
+ Arguments:
+ * entries - An array of key/value structs, where the first field is a non-null string key.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(array(struct('a', 1), struct('b', 2)));
+ {"a":1,"b":2}
+ """,
+ since = "4.4.0",
+ group = "variant_funcs")
+// scalastyle:on line.size.limit
+case class VariantFromEntries(child: Expression)
+ extends UnaryExpression
+ with QueryErrorsBase {
+ override def nullIntolerant: Boolean = true
+
+ @transient
+ private lazy val dataTypeDetails: Option[(DataType, Boolean)] = child.dataType match {
+ case ArrayType(
+ StructType(Array(StructField(_, _, _, _), StructField(_, valueType, _, _))),
+ containsNull) =>
+ Some((valueType, containsNull))
+ case _ => None
+ }
+
+ @transient private lazy val valueType: DataType = dataTypeDetails.get._1
+ @transient private lazy val nullEntries: Boolean = dataTypeDetails.get._2
+
+ override def nullable: Boolean = child.nullable || nullEntries
+ override def dataType: DataType = VariantType
+
+ override def checkInputDataTypes(): TypeCheckResult = child.dataType match {
+ case ArrayType(
+ StructType(Array(StructField(_, _: StringType, _, _), StructField(_, vt, _, _))), _) =>
+ if (VariantGet.checkDataType(vt, allowStructsAndMaps = true)) {
+ TypeCheckResult.TypeCheckSuccess
+ } else {
+ DataTypeMismatch(
+ errorSubClass = "CAST_WITHOUT_SUGGESTION",
+ messageParameters =
+ Map("srcType" -> toSQLType(vt), "targetType" -> toSQLType(VariantType)))
+ }
+ case _ =>
+ DataTypeMismatch(
+ errorSubClass = "UNEXPECTED_INPUT_TYPE",
+ messageParameters = Map(
+ "paramIndex" -> ordinalNumber(0),
+ "requiredType" -> s"${toSQLType(ArrayType)} of pair ${toSQLType(StructType)}",
+ "inputSql" -> toSQLExpr(child),
+ "inputType" -> toSQLType(child.dataType)))
+ }
+
+ override def prettyName: String = "variant_from_entries"
+
+ override protected def withNewChildInternal(newChild: Expression): VariantFromEntries =
+ copy(child = newChild)
+
+ override protected def nullSafeEval(input: Any): Any = {
+ val entries = input.asInstanceOf[ArrayData]
+ VariantExpressionEvalUtils.variantFromEntries(entries, valueType)
+ }
+
+ override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+ nullSafeCodeGen(ctx, ev, c => {
+ val cls = variant.VariantExpressionEvalUtils.getClass.getName.stripSuffix("$")
+ val valueTypeArg = ctx.addReferenceObj("valueType", valueType)
+ // nullSafeCodeGen only declares `ev.isNull` as a local when this expression is
+ // nullable; when it isn't, entries can never contain a null and the helper can never
+ // return null, so the reassignment must be skipped rather than referencing an
+ // undeclared variable.
+ val markNull = if (nullable) s"${ev.isNull} = ${ev.value} == null;" else ""
+ s"""
+ |${ev.value} = $cls.variantFromEntries($c, $valueTypeArg);
+ |$markNull
+ """.stripMargin
+ })
+ }
+}
+
// A path segment in the `VariantGet` expression represents either an object key access or an array
// index access.
sealed abstract class VariantPathSegment extends Serializable
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala
index b3248609ae3e..7f399faae242 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala
@@ -986,6 +986,70 @@ class VariantExpressionSuite extends SparkFunSuite with ExpressionEvalHelper {
checkFailure(Map(1 -> 1), toVariantObject = true)
}
+ test("variant_from_arrays and variant_from_entries") {
+ def keysValues(keys: Any, values: Any, valueType: DataType): VariantFromArrays =
+ VariantFromArrays(
+ Literal.create(keys, ArrayType(StringType)),
+ Literal.create(values, ArrayType(valueType)))
+
+ def entriesOf(entries: Any, valueType: DataType,
+ containsNull: Boolean = false): VariantFromEntries =
+ VariantFromEntries(Literal.create(entries, ArrayType(
+ StructType(Seq(StructField("k", StringType), StructField("v", valueType))), containsNull)))
+
+ // Basic object construction; keys are sorted in the resulting variant object.
+ checkEvaluation(StructsToJson(Map.empty,
+ keysValues(Array("z", "a"), Array(1, 2), IntegerType)), """{"a":2,"z":1}""")
+ checkEvaluation(StructsToJson(Map.empty,
+ entriesOf(Array(Row("a", 1), Row("b", 2)), IntegerType)), """{"a":1,"b":2}""")
+
+ // Empty input produces an empty object.
+ checkEvaluation(StructsToJson(Map.empty,
+ keysValues(Array.empty[String], Array.empty[Int], IntegerType)), "{}")
+
+ // Null values are kept as variant null; nested values are converted recursively.
+ checkEvaluation(StructsToJson(Map.empty,
+ entriesOf(Array(Row("a", 1), Row("b", null)), IntegerType)), """{"a":1,"b":null}""")
+ checkEvaluation(StructsToJson(Map.empty,
+ keysValues(Array("a"), Array(Array(1, 2, 3)), ArrayType(IntegerType))), """{"a":[1,2,3]}""")
+ checkEvaluation(StructsToJson(Map.empty, keysValues(Array("a"), Array(Row(1)),
+ StructType(Seq(StructField("i", IntegerType))))), """{"a":{"i":1}}""")
+
+ // A null entry makes the whole result null.
+ checkEvaluation(StructsToJson(Map.empty,
+ entriesOf(Array(Row("a", 1), null), IntegerType, containsNull = true)), null)
+
+ // A null entry dominates a value-conversion failure in an earlier entry (matches
+ // map_from_entries: the null check runs for every entry before any value is converted).
+ checkEvaluation(StructsToJson(Map.empty,
+ entriesOf(Array(Row("a", Row(1, 2)), null),
+ StructType(Seq(StructField("x", IntegerType), StructField("x", IntegerType))),
+ containsNull = true)), null)
+
+ // A null array input produces null.
+ checkEvaluation(StructsToJson(Map.empty, VariantFromArrays(
+ Literal.create(null, ArrayType(StringType)),
+ Literal.create(Array(1), ArrayType(IntegerType)))), null)
+
+ // A null key is rejected.
+ checkErrorInExpression[SparkRuntimeException](
+ keysValues(Array("a", null), Array(1, 2), IntegerType),
+ "NULL_MAP_KEY", Map.empty[String, String])
+
+ // Duplicate keys are rejected for both forms.
+ checkErrorInExpression[SparkRuntimeException](
+ keysValues(Array("a", "a"), Array(1, 2), IntegerType),
+ "VARIANT_DUPLICATE_KEY", Map("key" -> "a"))
+ checkErrorInExpression[SparkRuntimeException](
+ entriesOf(Array(Row("a", 1), Row("a", 2)), IntegerType),
+ "VARIANT_DUPLICATE_KEY", Map("key" -> "a"))
+
+ // Mismatched array lengths are rejected.
+ checkErrorInExpression[SparkRuntimeException](
+ keysValues(Array("a", "b"), Array(1), IntegerType),
+ "_LEGACY_ERROR_TEMP_2128", Map.empty[String, String])
+ }
+
test("schema_of_variant - unknown type") {
val emptyMetadata = Array[Byte](VERSION, 0, 0)
diff --git a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala
index 04074accf5c8..25a873d821ee 100644
--- a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala
+++ b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala
@@ -2827,6 +2827,14 @@ class PlanGenerationTestSuite extends ConnectFunSuite with Logging {
fn.schema_of_variant(fn.parse_json(fn.col("g")))
}
+ functionTest("variant_from_arrays") {
+ fn.variant_from_arrays(fn.array(lit("a"), lit("b")), fn.array(lit(1), lit(2)))
+ }
+
+ functionTest("variant_from_entries") {
+ fn.variant_from_entries(fn.array(fn.struct(lit("a"), lit(1)), fn.struct(lit("b"), lit(2))))
+ }
+
functionTest("schema_of_variant_agg") {
fn.schema_of_variant_agg(fn.parse_json(fn.col("g")))
}
diff --git a/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_from_arrays.explain b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_from_arrays.explain
new file mode 100644
index 000000000000..522406c52b06
--- /dev/null
+++ b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_from_arrays.explain
@@ -0,0 +1,2 @@
+Project [variant_from_arrays(array(a, b), array(1, 2)) AS variant_from_arrays(array(a, b), array(1, 2))#0]
++- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
diff --git a/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_from_entries.explain b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_from_entries.explain
new file mode 100644
index 000000000000..550854623fa9
--- /dev/null
+++ b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_from_entries.explain
@@ -0,0 +1,2 @@
+Project [variant_from_entries(array(struct(col1, a, col2, 1), struct(col1, b, col2, 2))) AS variant_from_entries(array(struct(a, 1), struct(b, 2)))#0]
++- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
diff --git a/sql/connect/common/src/test/resources/query-tests/queries/function_variant_from_arrays.json b/sql/connect/common/src/test/resources/query-tests/queries/function_variant_from_arrays.json
new file mode 100644
index 000000000000..1a9d2c5ae148
--- /dev/null
+++ b/sql/connect/common/src/test/resources/query-tests/queries/function_variant_from_arrays.json
@@ -0,0 +1,169 @@
+{
+ "common": {
+ "planId": "1"
+ },
+ "project": {
+ "input": {
+ "common": {
+ "planId": "0"
+ },
+ "localRelation": {
+ "schema": "struct\u003cid:bigint,a:int,b:double,d:struct\u003cid:bigint,a:int,b:double\u003e,e:array\u003cint\u003e,f:map\u003cstring,struct\u003cid:bigint,a:int,b:double\u003e\u003e,g:string\u003e"
+ }
+ },
+ "expressions": [{
+ "unresolvedFunction": {
+ "functionName": "variant_from_arrays",
+ "arguments": [{
+ "unresolvedFunction": {
+ "functionName": "array",
+ "arguments": [{
+ "literal": {
+ "string": "a"
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "lit",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }, {
+ "literal": {
+ "string": "b"
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "lit",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }],
+ "isInternal": false
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "array",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }, {
+ "unresolvedFunction": {
+ "functionName": "array",
+ "arguments": [{
+ "literal": {
+ "integer": 1
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "lit",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }, {
+ "literal": {
+ "integer": 2
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "lit",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }],
+ "isInternal": false
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "array",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }],
+ "isInternal": false
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "variant_from_arrays",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }]
+ }
+}
\ No newline at end of file
diff --git a/sql/connect/common/src/test/resources/query-tests/queries/function_variant_from_arrays.proto.bin b/sql/connect/common/src/test/resources/query-tests/queries/function_variant_from_arrays.proto.bin
new file mode 100644
index 000000000000..a005fd07d20e
Binary files /dev/null and b/sql/connect/common/src/test/resources/query-tests/queries/function_variant_from_arrays.proto.bin differ
diff --git a/sql/connect/common/src/test/resources/query-tests/queries/function_variant_from_entries.json b/sql/connect/common/src/test/resources/query-tests/queries/function_variant_from_entries.json
new file mode 100644
index 000000000000..d41b6b387bc3
--- /dev/null
+++ b/sql/connect/common/src/test/resources/query-tests/queries/function_variant_from_entries.json
@@ -0,0 +1,192 @@
+{
+ "common": {
+ "planId": "1"
+ },
+ "project": {
+ "input": {
+ "common": {
+ "planId": "0"
+ },
+ "localRelation": {
+ "schema": "struct\u003cid:bigint,a:int,b:double,d:struct\u003cid:bigint,a:int,b:double\u003e,e:array\u003cint\u003e,f:map\u003cstring,struct\u003cid:bigint,a:int,b:double\u003e\u003e,g:string\u003e"
+ }
+ },
+ "expressions": [{
+ "unresolvedFunction": {
+ "functionName": "variant_from_entries",
+ "arguments": [{
+ "unresolvedFunction": {
+ "functionName": "array",
+ "arguments": [{
+ "unresolvedFunction": {
+ "functionName": "struct",
+ "arguments": [{
+ "literal": {
+ "string": "a"
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "lit",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }, {
+ "literal": {
+ "integer": 1
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "lit",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }],
+ "isInternal": false
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "struct",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }, {
+ "unresolvedFunction": {
+ "functionName": "struct",
+ "arguments": [{
+ "literal": {
+ "string": "b"
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "lit",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }, {
+ "literal": {
+ "integer": 2
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "lit",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }],
+ "isInternal": false
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "struct",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }],
+ "isInternal": false
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "array",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }],
+ "isInternal": false
+ },
+ "common": {
+ "origin": {
+ "jvmOrigin": {
+ "stackTrace": [{
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.functions$",
+ "methodName": "variant_from_entries",
+ "fileName": "functions.scala"
+ }, {
+ "classLoaderName": "app",
+ "declaringClass": "org.apache.spark.sql.PlanGenerationTestSuite",
+ "methodName": "~~trimmed~anonfun~~",
+ "fileName": "PlanGenerationTestSuite.scala"
+ }]
+ }
+ }
+ }
+ }]
+ }
+}
\ No newline at end of file
diff --git a/sql/connect/common/src/test/resources/query-tests/queries/function_variant_from_entries.proto.bin b/sql/connect/common/src/test/resources/query-tests/queries/function_variant_from_entries.proto.bin
new file mode 100644
index 000000000000..09d0cc19b916
Binary files /dev/null and b/sql/connect/common/src/test/resources/query-tests/queries/function_variant_from_entries.proto.bin differ
diff --git a/sql/core/src/test/resources/sql-functions/sql-expression-schema.md b/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
index 3923f966ab23..c19cc871122d 100644
--- a/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
+++ b/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
@@ -568,6 +568,8 @@
| org.apache.spark.sql.catalyst.expressions.variant.TryVariantSetExpressionBuilder | try_variant_set | SELECT try_variant_set(parse_json('{"a": 1}'), '$.a', 2) | struct<try_variant_set(parse_json({"a": 1}), $.a, 2, true):variant> |
| org.apache.spark.sql.catalyst.expressions.variant.VariantArrayAppendExpressionBuilder | variant_array_append | SELECT variant_array_append(parse_json('[1, 2, 3]'), '$', 4) | struct<variant_array_append(parse_json([1, 2, 3]), $, 4):variant> |
| org.apache.spark.sql.catalyst.expressions.variant.VariantDelete | variant_delete | SELECT variant_delete(parse_json('{"a": 1, "b": 2, "c": 3, "items": [1, 2, 3]}'), NULL, '$.a', '$.c') | struct<variant_delete(parse_json({"a": 1, "b": 2, "c": 3, "items": [1, 2, 3]}), NULL, $.a, $.c):variant> |
+| org.apache.spark.sql.catalyst.expressions.variant.VariantFromArrays | variant_from_arrays | SELECT variant_from_arrays(array('a', 'b'), array(1, 2)) | struct<variant_from_arrays(array(a, b), array(1, 2)):variant> |
+| org.apache.spark.sql.catalyst.expressions.variant.VariantFromEntries | variant_from_entries | SELECT variant_from_entries(array(struct('a', 1), struct('b', 2))) | struct<variant_from_entries(array(struct(a, 1), struct(b, 2))):variant> |
| org.apache.spark.sql.catalyst.expressions.variant.VariantGetExpressionBuilder | variant_get | SELECT variant_get(parse_json('{"a": 1}'), '$.a', 'int') | struct<variant_get(parse_json({"a": 1}), $.a):int> |
| org.apache.spark.sql.catalyst.expressions.variant.VariantInsertExpressionBuilder | variant_insert | SELECT variant_insert(parse_json('{"a": 1}'), '$.b', 2) | struct<variant_insert(parse_json({"a": 1}), $.b, 2):variant> |
| org.apache.spark.sql.catalyst.expressions.variant.VariantSetExpressionBuilder | variant_set | SELECT variant_set(parse_json('{"a": 1}'), '$.a', 2) | struct<variant_set(parse_json({"a": 1}), $.a, 2, true):variant> |
diff --git a/sql/core/src/test/resources/sql-tests/analyzer-results/variant/variant-from-arrays-entries.sql.out b/sql/core/src/test/resources/sql-tests/analyzer-results/variant/variant-from-arrays-entries.sql.out
new file mode 100644
index 000000000000..0f159016698c
--- /dev/null
+++ b/sql/core/src/test/resources/sql-tests/analyzer-results/variant/variant-from-arrays-entries.sql.out
@@ -0,0 +1,244 @@
+-- Automatically generated by SQLQueryTestSuite
+-- !query
+select cast(variant_from_arrays(array('z', 'a'), array(1, 2)) as string)
+-- !query analysis
+Project [cast(variant_from_arrays(array(z, a), array(1, 2)) as string) AS CAST(variant_from_arrays(array(z, a), array(1, 2)) AS STRING)#x]
++- OneRowRelation
+
+
+-- !query
+select cast(variant_from_arrays(cast(array() as array<string>), cast(array() as array<int>)) as string)
+-- !query analysis
+Project [cast(variant_from_arrays(cast(array() as array<string>), cast(array() as array<int>)) as string) AS CAST(variant_from_arrays(array(), array()) AS STRING)#x]
++- OneRowRelation
+
+
+-- !query
+select cast(variant_from_arrays(array('a', 'b'), array(1, cast(null as int))) as string)
+-- !query analysis
+Project [cast(variant_from_arrays(array(a, b), array(1, cast(null as int))) as string) AS CAST(variant_from_arrays(array(a, b), array(1, CAST(NULL AS INT))) AS STRING)#x]
++- OneRowRelation
+
+
+-- !query
+select cast(variant_from_arrays(array('a'), array(array(1, 2, 3))) as string)
+-- !query analysis
+Project [cast(variant_from_arrays(array(a), array(array(1, 2, 3))) as string) AS CAST(variant_from_arrays(array(a), array(array(1, 2, 3))) AS STRING)#x]
++- OneRowRelation
+
+
+-- !query
+select cast(variant_from_arrays(cast(null as array<string>), array(1)) as string)
+-- !query analysis
+Project [cast(variant_from_arrays(cast(null as array<string>), array(1)) as string) AS CAST(variant_from_arrays(NULL, array(1)) AS STRING)#x]
++- OneRowRelation
+
+
+-- !query
+select variant_from_arrays(array('a', cast(null as string)), array(1, 2))
+-- !query analysis
+Project [variant_from_arrays(array(a, cast(null as string)), array(1, 2)) AS variant_from_arrays(array(a, CAST(NULL AS STRING)), array(1, 2))#x]
++- OneRowRelation
+
+
+-- !query
+select variant_from_arrays(array('a', 'a'), array(1, 2))
+-- !query analysis
+Project [variant_from_arrays(array(a, a), array(1, 2)) AS variant_from_arrays(array(a, a), array(1, 2))#x]
++- OneRowRelation
+
+
+-- !query
+select variant_from_arrays(array('a', 'b'), array(1))
+-- !query analysis
+Project [variant_from_arrays(array(a, b), array(1)) AS variant_from_arrays(array(a, b), array(1))#x]
++- OneRowRelation
+
+
+-- !query
+select variant_from_arrays(array(1, 2), array('a', 'b'))
+-- !query analysis
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "inputSql" : "\"array(1, 2)\"",
+ "inputType" : "\"ARRAY<INT>\"",
+ "paramIndex" : "first",
+ "requiredType" : "\"ARRAY<STRING>\"",
+ "sqlExpr" : "\"variant_from_arrays(array(1, 2), array(a, b))\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 8,
+ "stopIndex" : 56,
+ "fragment" : "variant_from_arrays(array(1, 2), array('a', 'b'))"
+ } ]
+}
+
+
+-- !query
+select variant_from_arrays(array('a'), array(map(1, 2)))
+-- !query analysis
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.CAST_WITHOUT_SUGGESTION",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "sqlExpr" : "\"variant_from_arrays(array(a), array(map(1, 2)))\"",
+ "srcType" : "\"MAP<INT, INT>\"",
+ "targetType" : "\"VARIANT\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 8,
+ "stopIndex" : 56,
+ "fragment" : "variant_from_arrays(array('a'), array(map(1, 2)))"
+ } ]
+}
+
+
+-- !query
+select cast(variant_from_entries(array(named_struct('k', 'a', 'v', 1), named_struct('k', 'b', 'v', 2))) as string)
+-- !query analysis
+Project [cast(variant_from_entries(array(named_struct(k, a, v, 1), named_struct(k, b, v, 2))) as string) AS CAST(variant_from_entries(array(named_struct(k, a, v, 1), named_struct(k, b, v, 2))) AS STRING)#x]
++- OneRowRelation
+
+
+-- !query
+select cast(variant_from_entries(cast(array() as array<struct<k:string,v:int>>)) as string)
+-- !query analysis
+Project [cast(variant_from_entries(cast(array() as array<struct<k:string,v:int>>)) as string) AS CAST(variant_from_entries(array()) AS STRING)#x]
++- OneRowRelation
+
+
+-- !query
+select cast(variant_from_entries(array(named_struct('k', 'a', 'v', cast(null as int)))) as string)
+-- !query analysis
+Project [cast(variant_from_entries(array(named_struct(k, a, v, cast(null as int)))) as string) AS CAST(variant_from_entries(array(named_struct(k, a, v, CAST(NULL AS INT)))) AS STRING)#x]
++- OneRowRelation
+
+
+-- !query
+select cast(variant_from_entries(array(named_struct('k', 'a', 'v', 1), cast(null as struct<k:string,v:int>))) as string)
+-- !query analysis
+Project [cast(variant_from_entries(array(named_struct(k, a, v, 1), cast(null as struct<k:string,v:int>))) as string) AS CAST(variant_from_entries(array(named_struct(k, a, v, 1), NULL)) AS STRING)#x]
++- OneRowRelation
+
+
+-- !query
+select cast(variant_from_entries(cast(null as array<struct<k:string,v:int>>)) as string)
+-- !query analysis
+Project [cast(variant_from_entries(cast(null as array<struct<k:string,v:int>>)) as string) AS CAST(variant_from_entries(NULL) AS STRING)#x]
++- OneRowRelation
+
+
+-- !query
+select variant_from_entries(array(named_struct('k', cast(null as string), 'v', 1)))
+-- !query analysis
+Project [variant_from_entries(array(named_struct(k, cast(null as string), v, 1))) AS variant_from_entries(array(named_struct(k, CAST(NULL AS STRING), v, 1)))#x]
++- OneRowRelation
+
+
+-- !query
+select variant_from_entries(array(named_struct('k', 'a', 'v', 1), named_struct('k', 'a', 'v', 2)))
+-- !query analysis
+Project [variant_from_entries(array(named_struct(k, a, v, 1), named_struct(k, a, v, 2))) AS variant_from_entries(array(named_struct(k, a, v, 1), named_struct(k, a, v, 2)))#x]
++- OneRowRelation
+
+
+-- !query
+select variant_from_entries(array(1, 2))
+-- !query analysis
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "inputSql" : "\"array(1, 2)\"",
+ "inputType" : "\"ARRAY<INT>\"",
+ "paramIndex" : "first",
+ "requiredType" : "\"ARRAY\" of pair \"STRUCT\"",
+ "sqlExpr" : "\"variant_from_entries(array(1, 2))\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 8,
+ "stopIndex" : 40,
+ "fragment" : "variant_from_entries(array(1, 2))"
+ } ]
+}
+
+
+-- !query
+select variant_from_entries(array(named_struct('k', 'a')))
+-- !query analysis
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "inputSql" : "\"array(named_struct(k, a))\"",
+ "inputType" : "\"ARRAY<STRUCT<k: STRING NOT NULL>>\"",
+ "paramIndex" : "first",
+ "requiredType" : "\"ARRAY\" of pair \"STRUCT\"",
+ "sqlExpr" : "\"variant_from_entries(array(named_struct(k, a)))\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 8,
+ "stopIndex" : 58,
+ "fragment" : "variant_from_entries(array(named_struct('k', 'a')))"
+ } ]
+}
+
+
+-- !query
+select variant_from_entries(array(named_struct('k', 1, 'v', 'a')))
+-- !query analysis
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "inputSql" : "\"array(named_struct(k, 1, v, a))\"",
+ "inputType" : "\"ARRAY<STRUCT<k: INT NOT NULL, v: STRING NOT NULL>>\"",
+ "paramIndex" : "first",
+ "requiredType" : "\"ARRAY\" of pair \"STRUCT\"",
+ "sqlExpr" : "\"variant_from_entries(array(named_struct(k, 1, v, a)))\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 8,
+ "stopIndex" : 66,
+ "fragment" : "variant_from_entries(array(named_struct('k', 1, 'v', 'a')))"
+ } ]
+}
+
+
+-- !query
+select variant_from_entries(array(named_struct('k', 'a', 'v', map(1, 2))))
+-- !query analysis
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.CAST_WITHOUT_SUGGESTION",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "sqlExpr" : "\"variant_from_entries(array(named_struct(k, a, v, map(1, 2))))\"",
+ "srcType" : "\"MAP<INT, INT>\"",
+ "targetType" : "\"VARIANT\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 8,
+ "stopIndex" : 74,
+ "fragment" : "variant_from_entries(array(named_struct('k', 'a', 'v', map(1, 2))))"
+ } ]
+}
diff --git a/sql/core/src/test/resources/sql-tests/inputs/variant/variant-from-arrays-entries.sql b/sql/core/src/test/resources/sql-tests/inputs/variant/variant-from-arrays-entries.sql
new file mode 100644
index 000000000000..d4db88252d1f
--- /dev/null
+++ b/sql/core/src/test/resources/sql-tests/inputs/variant/variant-from-arrays-entries.sql
@@ -0,0 +1,47 @@
+-- variant_from_arrays
+
+-- Basic object construction; keys are sorted in the resulting variant object.
+select cast(variant_from_arrays(array('z', 'a'), array(1, 2)) as string);
+-- Empty input produces an empty object.
+select cast(variant_from_arrays(cast(array() as array<string>), cast(array() as array<int>)) as string);
+-- Null values are kept as variant null.
+select cast(variant_from_arrays(array('a', 'b'), array(1, cast(null as int))) as string);
+-- Nested values are converted recursively.
+select cast(variant_from_arrays(array('a'), array(array(1, 2, 3))) as string);
+-- A null array input produces null.
+select cast(variant_from_arrays(cast(null as array<string>), array(1)) as string);
+-- A null key is rejected.
+select variant_from_arrays(array('a', cast(null as string)), array(1, 2));
+-- Duplicate keys are rejected.
+select variant_from_arrays(array('a', 'a'), array(1, 2));
+-- Mismatched array lengths are rejected.
+select variant_from_arrays(array('a', 'b'), array(1));
+-- A non-string key type is rejected.
+select variant_from_arrays(array(1, 2), array('a', 'b'));
+-- A value type that cannot be cast to variant is rejected.
+select variant_from_arrays(array('a'), array(map(1, 2)));
+
+-- variant_from_entries
+
+-- Basic object construction from key/value struct entries.
+select cast(variant_from_entries(array(named_struct('k', 'a', 'v', 1), named_struct('k', 'b', 'v', 2))) as string);
+-- Empty input produces an empty object.
+select cast(variant_from_entries(cast(array() as array<struct<k:string,v:int>>)) as string);
+-- Null values are kept as variant null.
+select cast(variant_from_entries(array(named_struct('k', 'a', 'v', cast(null as int)))) as string);
+-- A null entry makes the whole result null.
+select cast(variant_from_entries(array(named_struct('k', 'a', 'v', 1), cast(null as struct<k:string,v:int>))) as string);
+-- A null array input produces null.
+select cast(variant_from_entries(cast(null as array<struct<k:string,v:int>>)) as string);
+-- A null key is rejected.
+select variant_from_entries(array(named_struct('k', cast(null as string), 'v', 1)));
+-- Duplicate keys are rejected.
+select variant_from_entries(array(named_struct('k', 'a', 'v', 1), named_struct('k', 'a', 'v', 2)));
+-- A non-array-of-pair-struct input is rejected.
+select variant_from_entries(array(1, 2));
+-- A struct with the wrong number of fields is rejected.
+select variant_from_entries(array(named_struct('k', 'a')));
+-- A non-string key type is rejected.
+select variant_from_entries(array(named_struct('k', 1, 'v', 'a')));
+-- A value type that cannot be cast to variant is rejected.
+select variant_from_entries(array(named_struct('k', 'a', 'v', map(1, 2))));
diff --git a/sql/core/src/test/resources/sql-tests/results/variant/variant-from-arrays-entries.sql.out b/sql/core/src/test/resources/sql-tests/results/variant/variant-from-arrays-entries.sql.out
new file mode 100644
index 000000000000..c51ef9d03a0b
--- /dev/null
+++ b/sql/core/src/test/resources/sql-tests/results/variant/variant-from-arrays-entries.sql.out
@@ -0,0 +1,296 @@
+-- Automatically generated by SQLQueryTestSuite
+-- !query
+select cast(variant_from_arrays(array('z', 'a'), array(1, 2)) as string)
+-- !query schema
+struct<CAST(variant_from_arrays(array(z, a), array(1, 2)) AS STRING):string>
+-- !query output
+{"a":2,"z":1}
+
+
+-- !query
+select cast(variant_from_arrays(cast(array() as array<string>), cast(array() as array<int>)) as string)
+-- !query schema
+struct<CAST(variant_from_arrays(array(), array()) AS STRING):string>
+-- !query output
+{}
+
+
+-- !query
+select cast(variant_from_arrays(array('a', 'b'), array(1, cast(null as int))) as string)
+-- !query schema
+struct<CAST(variant_from_arrays(array(a, b), array(1, CAST(NULL AS INT))) AS STRING):string>
+-- !query output
+{"a":1,"b":null}
+
+
+-- !query
+select cast(variant_from_arrays(array('a'), array(array(1, 2, 3))) as string)
+-- !query schema
+struct<CAST(variant_from_arrays(array(a), array(array(1, 2, 3))) AS STRING):string>
+-- !query output
+{"a":[1,2,3]}
+
+
+-- !query
+select cast(variant_from_arrays(cast(null as array<string>), array(1)) as string)
+-- !query schema
+struct<CAST(variant_from_arrays(NULL, array(1)) AS STRING):string>
+-- !query output
+NULL
+
+
+-- !query
+select variant_from_arrays(array('a', cast(null as string)), array(1, 2))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkRuntimeException
+{
+ "errorClass" : "NULL_MAP_KEY",
+ "sqlState" : "2200E"
+}
+
+
+-- !query
+select variant_from_arrays(array('a', 'a'), array(1, 2))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkRuntimeException
+{
+ "errorClass" : "VARIANT_DUPLICATE_KEY",
+ "sqlState" : "22023",
+ "messageParameters" : {
+ "key" : "a"
+ }
+}
+
+
+-- !query
+select variant_from_arrays(array('a', 'b'), array(1))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkRuntimeException
+{
+ "errorClass" : "_LEGACY_ERROR_TEMP_2128"
+}
+
+
+-- !query
+select variant_from_arrays(array(1, 2), array('a', 'b'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "inputSql" : "\"array(1, 2)\"",
+ "inputType" : "\"ARRAY<INT>\"",
+ "paramIndex" : "first",
+ "requiredType" : "\"ARRAY<STRING>\"",
+ "sqlExpr" : "\"variant_from_arrays(array(1, 2), array(a, b))\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 8,
+ "stopIndex" : 56,
+ "fragment" : "variant_from_arrays(array(1, 2), array('a', 'b'))"
+ } ]
+}
+
+
+-- !query
+select variant_from_arrays(array('a'), array(map(1, 2)))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.CAST_WITHOUT_SUGGESTION",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "sqlExpr" : "\"variant_from_arrays(array(a), array(map(1, 2)))\"",
+ "srcType" : "\"MAP<INT, INT>\"",
+ "targetType" : "\"VARIANT\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 8,
+ "stopIndex" : 56,
+ "fragment" : "variant_from_arrays(array('a'), array(map(1, 2)))"
+ } ]
+}
+
+
+-- !query
+select cast(variant_from_entries(array(named_struct('k', 'a', 'v', 1), named_struct('k', 'b', 'v', 2))) as string)
+-- !query schema
+struct<CAST(variant_from_entries(array(named_struct(k, a, v, 1), named_struct(k, b, v, 2))) AS STRING):string>
+-- !query output
+{"a":1,"b":2}
+
+
+-- !query
+select cast(variant_from_entries(cast(array() as array<struct<k:string,v:int>>)) as string)
+-- !query schema
+struct<CAST(variant_from_entries(array()) AS STRING):string>
+-- !query output
+{}
+
+
+-- !query
+select cast(variant_from_entries(array(named_struct('k', 'a', 'v', cast(null as int)))) as string)
+-- !query schema
+struct<CAST(variant_from_entries(array(named_struct(k, a, v, CAST(NULL AS INT)))) AS STRING):string>
+-- !query output
+{"a":null}
+
+
+-- !query
+select cast(variant_from_entries(array(named_struct('k', 'a', 'v', 1), cast(null as struct<k:string,v:int>))) as string)
+-- !query schema
+struct<CAST(variant_from_entries(array(named_struct(k, a, v, 1), NULL)) AS STRING):string>
+-- !query output
+NULL
+
+
+-- !query
+select cast(variant_from_entries(cast(null as array<struct<k:string,v:int>>)) as string)
+-- !query schema
+struct<CAST(variant_from_entries(NULL) AS STRING):string>
+-- !query output
+NULL
+
+
+-- !query
+select variant_from_entries(array(named_struct('k', cast(null as string), 'v', 1)))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkRuntimeException
+{
+ "errorClass" : "NULL_MAP_KEY",
+ "sqlState" : "2200E"
+}
+
+
+-- !query
+select variant_from_entries(array(named_struct('k', 'a', 'v', 1), named_struct('k', 'a', 'v', 2)))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkRuntimeException
+{
+ "errorClass" : "VARIANT_DUPLICATE_KEY",
+ "sqlState" : "22023",
+ "messageParameters" : {
+ "key" : "a"
+ }
+}
+
+
+-- !query
+select variant_from_entries(array(1, 2))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "inputSql" : "\"array(1, 2)\"",
+ "inputType" : "\"ARRAY<INT>\"",
+ "paramIndex" : "first",
+ "requiredType" : "\"ARRAY\" of pair \"STRUCT\"",
+ "sqlExpr" : "\"variant_from_entries(array(1, 2))\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 8,
+ "stopIndex" : 40,
+ "fragment" : "variant_from_entries(array(1, 2))"
+ } ]
+}
+
+
+-- !query
+select variant_from_entries(array(named_struct('k', 'a')))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "inputSql" : "\"array(named_struct(k, a))\"",
+ "inputType" : "\"ARRAY<STRUCT<k: STRING NOT NULL>>\"",
+ "paramIndex" : "first",
+ "requiredType" : "\"ARRAY\" of pair \"STRUCT\"",
+ "sqlExpr" : "\"variant_from_entries(array(named_struct(k, a)))\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 8,
+ "stopIndex" : 58,
+ "fragment" : "variant_from_entries(array(named_struct('k', 'a')))"
+ } ]
+}
+
+
+-- !query
+select variant_from_entries(array(named_struct('k', 1, 'v', 'a')))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "inputSql" : "\"array(named_struct(k, 1, v, a))\"",
+ "inputType" : "\"ARRAY<STRUCT<k: INT NOT NULL, v: STRING NOT NULL>>\"",
+ "paramIndex" : "first",
+ "requiredType" : "\"ARRAY\" of pair \"STRUCT\"",
+ "sqlExpr" : "\"variant_from_entries(array(named_struct(k, 1, v, a)))\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 8,
+ "stopIndex" : 66,
+ "fragment" : "variant_from_entries(array(named_struct('k', 1, 'v', 'a')))"
+ } ]
+}
+
+
+-- !query
+select variant_from_entries(array(named_struct('k', 'a', 'v', map(1, 2))))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.CAST_WITHOUT_SUGGESTION",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "sqlExpr" : "\"variant_from_entries(array(named_struct(k, a, v, map(1, 2))))\"",
+ "srcType" : "\"MAP<INT, INT>\"",
+ "targetType" : "\"VARIANT\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 8,
+ "stopIndex" : 74,
+ "fragment" : "variant_from_entries(array(named_struct('k', 'a', 'v', map(1, 2))))"
+ } ]
+}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/VariantEndToEndSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/VariantEndToEndSuite.scala
index 2d26356890d2..30675e759428 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/VariantEndToEndSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/VariantEndToEndSuite.scala
@@ -246,6 +246,31 @@ class VariantEndToEndSuite extends SharedSparkSession {
}
}
+ test("variant_from_arrays and variant_from_entries - Codegen Support") {
+ Seq("CODEGEN_ONLY", "NO_CODEGEN").foreach { codegenMode =>
+ withSQLConf(SQLConf.CODEGEN_FACTORY_MODE.key -> codegenMode) {
+ val entryType = StructType(Array(
+ StructField("k", StringType), StructField("v", IntegerType)))
+ val schema = StructType(Array(
+ StructField("keys", ArrayType(StringType)),
+ StructField("values", ArrayType(IntegerType)),
+ StructField("entries", ArrayType(entryType))))
+ // Source non-foldable rows so the operators' doGenCode is actually exercised under codegen.
+ val data = Seq(Row(Seq("a", "b"), Seq(1, 2), Seq(Row("a", 1), Row("b", 2))))
+ val df = spark.createDataFrame(spark.sparkContext.parallelize(data), schema)
+ val arraysDF = df.select(variant_from_arrays(col("keys"), col("values")).cast("string"))
+ val entriesDF = df.select(variant_from_entries(col("entries")).cast("string"))
+ val wholeStage = codegenMode == "CODEGEN_ONLY"
+ assert(arraysDF.queryExecution.executedPlan.exists(
+ _.isInstanceOf[WholeStageCodegenExec]) == wholeStage)
+ assert(entriesDF.queryExecution.executedPlan.exists(
+ _.isInstanceOf[WholeStageCodegenExec]) == wholeStage)
+ checkAnswer(arraysDF, Row("""{"a":1,"b":2}"""))
+ checkAnswer(entriesDF, Row("""{"a":1,"b":2}"""))
+ }
+ }
+ }
+
test("schema_of_variant") {
def check(json: String, expected: String): Unit = {
val df = Seq(json).toDF("j").selectExpr("schema_of_variant(parse_json(j))")
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/collation/CollationExpressionWalkerSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/collation/CollationExpressionWalkerSuite.scala
index afe122f21226..ec5be2c60261 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/collation/CollationExpressionWalkerSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/collation/CollationExpressionWalkerSuite.scala
@@ -394,7 +394,11 @@ class CollationExpressionWalkerSuite extends SharedSparkSession {
"approx_top_k",
"approx_top_k_accumulate",
"approx_top_k_combine",
- "approx_top_k_estimate"
+ "approx_top_k_estimate",
+ // The variant object embeds the key string, which preserves the input case for collated
+ // strings, so the result is not comparable across collations.
+ "variant_from_arrays",
+ "variant_from_entries"
)
logInfo("Total number of expression: " + expressionCounter)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org