You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/07 23:19:50 UTC
(datafusion-comet) branch gh-readonly-queue/main/pr-6471-e76c35f1e782b78525d447ba09d6f580f9bf6dec created (now f80042f787)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-6471-e76c35f1e782b78525d447ba09d6f580f9bf6dec
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
at f80042f787 fix: Route collated array element comparisons through the codegen dispatcher (#6471)
This branch includes the following new commits:
new f80042f787 fix: Route collated array element comparisons through the codegen dispatcher (#6471)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: fix: Route collated array element comparisons through the codegen dispatcher (#6471)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6471-e76c35f1e782b78525d447ba09d6f580f9bf6dec
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit f80042f78715e5a009352f1582f6c246a5f96131
Author: Liang-Chi Hsieh <vi...@gmail.com>
AuthorDate: Wed Oct 7 23:19:26 2026 +0000
fix: Route collated array element comparisons through the codegen dispatcher (#6471)
* fix: Route collated array element comparisons through the codegen dispatcher
array_contains, arrays_overlap, array_distinct and array_union compare
array elements by raw bytes in their native kernels, so they returned
wrong results for strings with a non-UTF8_BINARY collation (for example
UTF8_LCASE or UTF8_BINARY_RTRIM) when a dispatched collated cast fed
them.
Report these cases as Incompatible and mix in CodegenDispatchFallback so
they run Spark's collation-aware comparison through the JVM codegen
dispatcher by default. ArrayDistinct gets its own serde for this.
* docs: Mark collation-routed array functions as Hybrid in expressions.md
---
docs/source/user-guide/latest/expressions.md | 8 +-
.../main/scala/org/apache/comet/serde/arrays.scala | 68 ++++++++++++++--
.../array/array_element_equality_collation.sql | 90 ++++++++++++++++++++++
...ement_equality_collation_allow_incompatible.sql | 45 +++++++++++
.../array_element_equality_collation_disabled.sql | 49 ++++++++++++
.../org/apache/comet/CometMapExpressionSuite.scala | 24 +++---
6 files changed, 265 insertions(+), 19 deletions(-)
diff --git a/docs/source/user-guide/latest/expressions.md b/docs/source/user-guide/latest/expressions.md
index 74232aac65..dffd6b4988 100644
--- a/docs/source/user-guide/latest/expressions.md
+++ b/docs/source/user-guide/latest/expressions.md
@@ -155,8 +155,8 @@ The tables below list every Spark built-in expression with its current status.
| `array` | ✅ | Native | |
| `array_append` | ✅ | Native | |
| `array_compact` | ✅ | — | |
-| `array_contains` | ✅ | Native | Float/double element arrays route through the JVM codegen dispatcher by default; the native path is opt-in via allowIncompatible |
-| `array_distinct` | ✅ | Native | Floating-point elements fall back on Spark versions other than 4.2.0; signed-zero and NaN results may differ with native opt-in ([details](compatibility/floating-point.md)) |
+| `array_contains` | ✅ | Hybrid | Float/double element arrays and non-default string collations route through the JVM codegen dispatcher by default; the native path is opt-in via allowIncompatible |
+| `array_distinct` | ✅ | Native | Floating-point elements fall back on Spark versions other than 4.2.0; signed-zero and NaN results may differ with native opt-in ([details](compatibility/floating-point.md)). Non-default string collations fall back |
| `array_except` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) |
| `array_insert` | ✅ | Native | |
| `array_intersect` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) |
@@ -167,8 +167,8 @@ The tables below list every Spark built-in expression with its current status.
| `array_prepend` | ✅ | — | |
| `array_remove` | ✅ | Native | |
| `array_repeat` | ✅ | Native | |
-| `array_union` | ✅ | Native | Floating-point elements fall back on Spark versions other than 4.2.0; signed-zero and NaN results may differ with native opt-in ([details](compatibility/floating-point.md)) |
-| `arrays_overlap` | ✅ | Native | |
+| `array_union` | ✅ | Native | Floating-point elements fall back on Spark versions other than 4.2.0; signed-zero and NaN results may differ with native opt-in ([details](compatibility/floating-point.md)). Non-default string collations fall back |
+| `arrays_overlap` | ✅ | Hybrid | Non-default string collations use the JVM codegen dispatcher |
| `arrays_zip` | ✅ | Native | |
| `element_at` | ✅ | Native | |
| `flatten` | ✅ | Native | Binary/struct/map elements fall back |
diff --git a/spark/src/main/scala/org/apache/comet/serde/arrays.scala b/spark/src/main/scala/org/apache/comet/serde/arrays.scala
index b2a75f8478..5aeb682434 100644
--- a/spark/src/main/scala/org/apache/comet/serde/arrays.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/arrays.scala
@@ -115,15 +115,24 @@ object CometArrayAppend extends CometExpressionSerde[ArrayAppend] with ArraysBas
object CometArrayContains
extends CometExpressionSerde[ArrayContains]
+ with CometTypeShim
with CodegenDispatchFallback {
+ override def hasConditionalNativeDefault: Boolean = true
+
private val floatingPointReason: String =
"Spark compares array elements with ordering.equiv, so -0.0 matches +0.0 and all NaNs match " +
"each other; Comet's native array_contains compares the raw Arrow values bitwise"
- override def getIncompatibleReasons(): Seq[String] = Seq(floatingPointReason)
+ private val collationReason: String =
+ ArrayElementEqualitySupport.collationReason("array_contains")
+
+ override def getIncompatibleReasons(): Seq[String] = Seq(floatingPointReason, collationReason)
override def getSupportLevel(expr: ArrayContains): SupportLevel = expr.left.dataType match {
+ // See ArrayElementEqualitySupport: bytewise string comparison ignores the collation.
+ case ArrayType(elementType, _) if hasNonDefaultStringCollation(elementType) =>
+ Incompatible(Some(collationReason))
// Native array_contains compares floating-point elements bitwise, disagreeing with Spark for
// -0.0/+0.0 and NaN. Report Incompatible (not Unsupported) for float/double element types (at
// any nesting level) so the expression routes through the JVM codegen dispatcher (Spark's own
@@ -309,7 +318,19 @@ object CometArrayMin extends CometExpressionSerde[ArrayMin] with CodegenDispatch
}
}
-object CometArraysOverlap extends CometExpressionSerde[ArraysOverlap] {
+object CometArraysOverlap
+ extends CometExpressionSerde[ArraysOverlap]
+ with CodegenDispatchFallback {
+
+ override def hasConditionalNativeDefault: Boolean = true
+
+ override def getIncompatibleReasons(): Seq[String] =
+ Seq(ArrayElementEqualitySupport.collationReason("arrays_overlap"))
+
+ // Both inputs share one element type, so checking the left side is enough.
+ override def getSupportLevel(expr: ArraysOverlap): SupportLevel =
+ ArrayElementEqualitySupport.getSupportLevel("arrays_overlap", expr.left.dataType)
+
override def convert(
expr: ArraysOverlap,
inputs: Seq[Attribute],
@@ -538,6 +559,30 @@ object CometSlice extends CometExpressionSerde[Slice] {
}
}
+/**
+ * Support level for native array kernels that compare elements for equality (membership, overlap,
+ * dedup). They compare strings by raw bytes, so a non-UTF8_BINARY collation (for example
+ * UTF8_LCASE, where 'a' equals 'A') gives wrong answers, and these cases report Incompatible.
+ * Serdes with a boolean result mix in CodegenDispatchFallback, so the JVM codegen dispatcher runs
+ * Spark's collation-aware comparison. Array-valued serdes fall back instead (see
+ * ArraySetSupport).
+ */
+private object ArrayElementEqualitySupport extends CometTypeShim {
+ def collationReason(name: String): String =
+ "Spark compares non-UTF8_BINARY collated string elements under their collation, while " +
+ s"Comet's native $name compares raw bytes"
+
+ def collationSupportLevel(name: String, arrayType: DataType): Option[SupportLevel] =
+ if (hasNonDefaultStringCollation(arrayType)) {
+ Some(Incompatible(Some(collationReason(name))))
+ } else {
+ None
+ }
+
+ def getSupportLevel(name: String, arrayType: DataType): SupportLevel =
+ collationSupportLevel(name, arrayType).getOrElse(Compatible())
+}
+
private[comet] object ArraySetSupport {
val floatingPointReason: String =
"Floating-point elements match Spark's signed-zero and NaN semantics natively only on " +
@@ -560,22 +605,33 @@ private[comet] object ArraySetSupport {
Compatible()
}
}
+
+ /** Collated string elements are also Incompatible; see ArrayElementEqualitySupport. */
+ def supportLevel(name: String, dataType: DataType): SupportLevel =
+ ArrayElementEqualitySupport
+ .collationSupportLevel(name, dataType)
+ .getOrElse(supportLevel(dataType))
+
+ def incompatibleReasons(name: String): Seq[String] =
+ Seq(floatingPointReason, ArrayElementEqualitySupport.collationReason(name))
}
// Use projection fallback to avoid codegen dispatch overhead for array-valued results.
// The native implementation remains available through opt-in.
object CometArrayDistinct extends CometScalarFunction[ArrayDistinct]("array_distinct") {
- override def getIncompatibleReasons(): Seq[String] = Seq(ArraySetSupport.floatingPointReason)
+ override def getIncompatibleReasons(): Seq[String] =
+ ArraySetSupport.incompatibleReasons("array_distinct")
override def getSupportLevel(expr: ArrayDistinct): SupportLevel =
- ArraySetSupport.supportLevel(expr.dataType)
+ ArraySetSupport.supportLevel("array_distinct", expr.dataType)
}
object CometArrayUnion extends CometExpressionSerde[ArrayUnion] {
- override def getIncompatibleReasons(): Seq[String] = Seq(ArraySetSupport.floatingPointReason)
+ override def getIncompatibleReasons(): Seq[String] =
+ ArraySetSupport.incompatibleReasons("array_union")
override def getSupportLevel(expr: ArrayUnion): SupportLevel =
- ArraySetSupport.supportLevel(expr.dataType)
+ ArraySetSupport.supportLevel("array_union", expr.dataType)
override def convert(
expr: ArrayUnion,
diff --git a/spark/src/test/resources/sql-tests/expressions/array/array_element_equality_collation.sql b/spark/src/test/resources/sql-tests/expressions/array/array_element_equality_collation.sql
new file mode 100644
index 0000000000..e807346182
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/array/array_element_equality_collation.sql
@@ -0,0 +1,90 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- MinSparkVersion: 4.0
+-- Config: spark.comet.exec.scalaUDF.codegen.enabled=true
+-- Config: spark.comet.expression.ArrayContains.allowIncompatible=false
+-- Config: spark.comet.expression.ArraysOverlap.allowIncompatible=false
+-- Config: spark.comet.expression.ArrayDistinct.allowIncompatible=false
+-- Config: spark.comet.expression.ArrayUnion.allowIncompatible=false
+
+-- array_contains, arrays_overlap, array_distinct and array_union compare array elements for
+-- equality. The native kernels compare strings by raw bytes, so under a non-UTF8_BINARY collation
+-- they would miss 'a' = 'A' (UTF8_LCASE) or 'x ' = 'x' (UTF8_BINARY_RTRIM). Collated
+-- array_contains and arrays_overlap route through the JVM codegen dispatcher, which runs Spark's
+-- collation-aware comparison. array_distinct and array_union return arrays, where dispatch costs
+-- more than a projection fallback (see ArraySetSupport), so they fall back to Spark instead.
+
+statement
+CREATE TABLE test_array_eq_collation(id int, a string, b string) USING parquet
+
+statement
+INSERT INTO test_array_eq_collation VALUES
+ (1, 'a', 'A'),
+ (2, 'x ', 'x'),
+ (3, 'b', 'c'),
+ (4, NULL, 'a'),
+ (5, 'A', NULL)
+
+-- UTF8_BINARY strings stay on the native kernels.
+query expect_native(array_contains,arrays_overlap,array_distinct,array_union)
+SELECT id, array_contains(array(a), b), arrays_overlap(array(a), array(b)),
+ array_distinct(array(a, b)), array_union(array(a), array(b))
+FROM test_array_eq_collation
+
+-- UTF8_LCASE: 'a' and 'A' are equal.
+query expect_dispatch(array_contains)
+SELECT id, array_contains(array(CAST(a AS STRING COLLATE UTF8_LCASE)), CAST(b AS STRING COLLATE UTF8_LCASE))
+FROM test_array_eq_collation
+
+query expect_dispatch(arrays_overlap)
+SELECT id, arrays_overlap(array(CAST(a AS STRING COLLATE UTF8_LCASE)), array(CAST(b AS STRING COLLATE UTF8_LCASE)))
+FROM test_array_eq_collation
+
+query expect_fallback(native array_distinct compares raw bytes)
+SELECT id, array_distinct(array(CAST(a AS STRING COLLATE UTF8_LCASE), CAST(b AS STRING COLLATE UTF8_LCASE)))
+FROM test_array_eq_collation
+
+query expect_fallback(native array_union compares raw bytes)
+SELECT id, array_union(array(CAST(a AS STRING COLLATE UTF8_LCASE)), array(CAST(b AS STRING COLLATE UTF8_LCASE)))
+FROM test_array_eq_collation
+
+-- UTF8_BINARY_RTRIM: 'x ' and 'x' are equal.
+query expect_dispatch(array_contains)
+SELECT id, array_contains(array(CAST(a AS STRING COLLATE UTF8_BINARY_RTRIM)), CAST(b AS STRING COLLATE UTF8_BINARY_RTRIM))
+FROM test_array_eq_collation
+
+query expect_dispatch(arrays_overlap)
+SELECT id, arrays_overlap(array(CAST(a AS STRING COLLATE UTF8_BINARY_RTRIM)), array(CAST(b AS STRING COLLATE UTF8_BINARY_RTRIM)))
+FROM test_array_eq_collation
+
+query expect_fallback(native array_distinct compares raw bytes)
+SELECT id, array_distinct(array(CAST(a AS STRING COLLATE UTF8_BINARY_RTRIM), CAST(b AS STRING COLLATE UTF8_BINARY_RTRIM)))
+FROM test_array_eq_collation
+
+query expect_fallback(native array_union compares raw bytes)
+SELECT id, array_union(array(CAST(a AS STRING COLLATE UTF8_BINARY_RTRIM)), array(CAST(b AS STRING COLLATE UTF8_BINARY_RTRIM)))
+FROM test_array_eq_collation
+
+-- Collation detection also recurses into struct fields of the array elements.
+query expect_dispatch(array_contains,arrays_overlap)
+SELECT id,
+ array_contains(array(named_struct('s', CAST(a AS STRING COLLATE UTF8_LCASE))),
+ named_struct('s', CAST(b AS STRING COLLATE UTF8_LCASE))),
+ arrays_overlap(array(named_struct('s', CAST(a AS STRING COLLATE UTF8_LCASE))),
+ array(named_struct('s', CAST(b AS STRING COLLATE UTF8_LCASE))))
+FROM test_array_eq_collation
diff --git a/spark/src/test/resources/sql-tests/expressions/array/array_element_equality_collation_allow_incompatible.sql b/spark/src/test/resources/sql-tests/expressions/array/array_element_equality_collation_allow_incompatible.sql
new file mode 100644
index 0000000000..7bedec72ce
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/array/array_element_equality_collation_allow_incompatible.sql
@@ -0,0 +1,45 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- MinSparkVersion: 4.0
+-- Config: spark.comet.exec.scalaUDF.codegen.enabled=true
+-- Config: spark.comet.expression.ArrayContains.allowIncompatible=true
+-- Config: spark.comet.expression.ArraysOverlap.allowIncompatible=true
+-- Config: spark.comet.expression.ArrayDistinct.allowIncompatible=true
+-- Config: spark.comet.expression.ArrayUnion.allowIncompatible=true
+
+-- allowIncompatible=true opts collated inputs into the native kernels, which compare raw bytes.
+-- The data only holds values whose bytewise and collation-aware equality agree, so the answers
+-- still match Spark; the point is that the native path is taken.
+
+statement
+CREATE TABLE test_array_eq_collation(id int, a string, b string) USING parquet
+
+statement
+INSERT INTO test_array_eq_collation VALUES (1, 'a', 'a'), (2, 'b', 'c'), (3, NULL, 'a')
+
+query expect_native(array_contains,arrays_overlap)
+SELECT id,
+ array_contains(array(CAST(a AS STRING COLLATE UTF8_LCASE)), CAST(b AS STRING COLLATE UTF8_LCASE)),
+ arrays_overlap(array(CAST(a AS STRING COLLATE UTF8_LCASE)), array(CAST(b AS STRING COLLATE UTF8_LCASE)))
+FROM test_array_eq_collation
+
+query expect_native(array_distinct,array_union)
+SELECT id,
+ array_distinct(array(CAST(a AS STRING COLLATE UTF8_LCASE), CAST(b AS STRING COLLATE UTF8_LCASE))),
+ array_union(array(CAST(a AS STRING COLLATE UTF8_LCASE)), array(CAST(b AS STRING COLLATE UTF8_LCASE)))
+FROM test_array_eq_collation
diff --git a/spark/src/test/resources/sql-tests/expressions/array/array_element_equality_collation_disabled.sql b/spark/src/test/resources/sql-tests/expressions/array/array_element_equality_collation_disabled.sql
new file mode 100644
index 0000000000..e07ad2f8a2
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/array/array_element_equality_collation_disabled.sql
@@ -0,0 +1,49 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- MinSparkVersion: 4.0
+-- Config: spark.comet.exec.scalaUDF.codegen.enabled=false
+-- Config: spark.comet.expression.ArrayContains.allowIncompatible=false
+-- Config: spark.comet.expression.ArraysOverlap.allowIncompatible=false
+-- Config: spark.comet.expression.ArrayDistinct.allowIncompatible=false
+-- Config: spark.comet.expression.ArrayUnion.allowIncompatible=false
+
+-- With the JVM codegen dispatcher disabled, collated array_contains and arrays_overlap have no
+-- Spark-compatible Comet path and fall back to Spark. array_distinct and array_union fall back
+-- whether or not the dispatcher is enabled.
+
+statement
+CREATE TABLE test_array_eq_collation(id int, a string, b string) USING parquet
+
+statement
+INSERT INTO test_array_eq_collation VALUES (1, 'a', 'A'), (2, 'x ', 'x'), (3, 'b', 'c'), (4, NULL, 'a')
+
+query expect_fallback(array_contains: spark.comet.exec.scalaUDF.codegen.enabled=false)
+SELECT id, array_contains(array(CAST(a AS STRING COLLATE UTF8_LCASE)), CAST(b AS STRING COLLATE UTF8_LCASE))
+FROM test_array_eq_collation
+
+query expect_fallback(arrays_overlap: spark.comet.exec.scalaUDF.codegen.enabled=false)
+SELECT id, arrays_overlap(array(CAST(a AS STRING COLLATE UTF8_BINARY_RTRIM)), array(CAST(b AS STRING COLLATE UTF8_BINARY_RTRIM)))
+FROM test_array_eq_collation
+
+query expect_fallback(native array_distinct compares raw bytes)
+SELECT id, array_distinct(array(CAST(a AS STRING COLLATE UTF8_LCASE), CAST(b AS STRING COLLATE UTF8_LCASE)))
+FROM test_array_eq_collation
+
+query expect_fallback(native array_union compares raw bytes)
+SELECT id, array_union(array(CAST(a AS STRING COLLATE UTF8_BINARY_RTRIM)), array(CAST(b AS STRING COLLATE UTF8_BINARY_RTRIM)))
+FROM test_array_eq_collation
diff --git a/spark/src/test/scala/org/apache/comet/CometMapExpressionSuite.scala b/spark/src/test/scala/org/apache/comet/CometMapExpressionSuite.scala
index 98ef07b96f..091f777dbe 100644
--- a/spark/src/test/scala/org/apache/comet/CometMapExpressionSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/CometMapExpressionSuite.scala
@@ -467,18 +467,24 @@ class CometMapExpressionSuite extends CometTestBase {
}
// The collated counterpart of the double-key `map_contains_key` case above: a nested
- // `UTF8_LCASE`-keyed map would reach `array_contains` with bytewise comparison. Expansion declines
- // the folded literal so the case-insensitive lookup stays on Spark. The outer lookup key is the
- // dynamic `_1` so the inner map survives as a literal (a constant key would let Spark fold
- // `map_keys` into a collated-string array literal instead, which never reaches this guard).
+ // `UTF8_LCASE`-keyed map would reach `array_contains` with bytewise comparison. Like the
+ // floating-point case, `CometArrayContains` reports collated element types Incompatible, so the
+ // default config codegen-dispatches it; `allowIncompatible=true` forces the native kernel so the
+ // expansion guard runs. Expansion declines the folded literal so the case-insensitive lookup
+ // stays on Spark. The outer lookup key is the dynamic `_1` so the inner map survives as a literal
+ // (a constant key would let Spark fold `map_keys` into a collated-string array literal instead,
+ // which never reaches this guard).
test("map_contains_key over nested collated map keys falls back (multirow)") {
assume(isSpark40Plus)
+ val query = "SELECT _1 AS id, map_contains_key(" +
+ "element_at(map(1, map(CAST('A1' AS STRING COLLATE UTF8_LCASE), 7)), _1), " +
+ "CAST('a1' AS STRING COLLATE UTF8_LCASE)) AS present FROM tbl"
withParquetTable((1 until 4).map(i => (i, i.toLong)), "tbl") {
- checkSparkAnswerAndFallbackReason(
- "SELECT _1 AS id, map_contains_key(" +
- "element_at(map(1, map(CAST('A1' AS STRING COLLATE UTF8_LCASE), 7)), _1), " +
- "CAST('a1' AS STRING COLLATE UTF8_LCASE)) AS present FROM tbl",
- "Unsupported data type MapType")
+ withSQLConf(CometConf.getExprAllowIncompatConfigKey(classOf[ArrayContains]) -> "true") {
+ checkSparkAnswerAndFallbackReason(query, "Unsupported data type MapType")
+ }
+ // Under the default config the collated lookup is dispatched and matches Spark.
+ checkSparkAnswer(query)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org