You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/09/09 14:28:20 UTC

[PR] perf: vectorize the native map lookup behind element_at and GetMapValue [datafusion-comet]

andygrove opened a new pull request, #5806:
URL: https://github.com/apache/datafusion-comet/pull/5806

   ## Which issue does this PR close?
   
   Closes #5795.
   
   ## Rationale for this change
   
   `GetMapValue` (`m[k]`) and `element_at(<map>, k)` both serialise to the DataFusion `map_extract`
   UDF, and the planner then unwrapped its one-element list result with a second `ListExtract` pass.
   
   The diagnosis in the issue holds. `general_map_extract_inner` re-slices the query key and every
   candidate key into a fresh `ArrayRef` per comparison and compares them through `dyn Array`
   equality:
   
   ```rust
   let query_key = query_keys_array.slice(row_index, 1);
   let value_index = (0..len).find(|&i| keys.slice(start + i, 1).as_ref() == query_key.as_ref());
   ```
   
   so the lookup is O(rows x entries-per-row) allocations, and the constant key is not hoisted out of
   the row loop. That is why the map lookup was the one map operation where Comet lost to Spark.
   
   ## What changes are included in this PR?
   
   A native `SparkMapExtract`, registered under the same `map_extract` name so it overrides the
   DataFusion one, in `native/spark-expr/src/map_funcs/map_extract.rs`:
   
   - one Arrow `eq` over the whole batch of map entries (broadcasting the key when it is a constant),
     then a bitmask scan for each row's first match, then one `take` to gather the values;
   - it returns the matched value directly instead of a one-element list, so the `ListExtract` wrapper
     in `planner.rs` is no longer needed and that second pass goes away;
   - a sliced `MapArray` keeps its original entry offsets, so only the visible entry window is
     compared.
   
   Semantics are unchanged: the first matching entry wins, and a missing key, a `NULL` map row and a
   `NULL` lookup key all yield `NULL`. Key types whose Spark equality a native lookup cannot reproduce
   (floating point, non-default collations, complex keys) are still declined by `MapKeySupport`, so
   they never reach the kernel. Arrow's `eq` rejects nested key types, so DataFusion's element-wise
   comparison is kept as a backstop for those, and a lookup key whose runtime type is not the map's
   key type is now rejected rather than silently missing every row.
   
   ## How are these changes tested?
   
   **Existing tests.** `CometMapExpressionSuite` (25), `CometArrayExpressionSuite` (64),
   `CometExpressionSuite` (141) and the full `CometSqlFileTestSuite` (474, including
   `element_at_map.sql`, `element_at_map_ansi.sql`, `element_at_map_collation.sql` and
   `get_map_value.sql`) all pass.
   
   **New tests.** 13 Rust unit tests covering hit/miss, duplicate keys, `NULL` maps, `NULL` values,
   `NULL` and per-row lookup keys, sliced maps, empty input, a scalar map argument, non-string keys,
   the nested-key backstop, and the rejected argument types. Two new `CometMapExpressionSuite` cases
   cover the same two tricky paths end to end: `element_at` under a native `OFFSET` (sliced map) and
   `element_at`/`m[k]` with a per-row lookup key.
   
   **Kernel benchmark** (new `native/spark-expr/benches/map_extract.rs`, 8192-row batches of
   `map<string, string>`, 10% `NULL` rows, 60 distinct keys, run against both implementations):
   
   | entries/map | case | `datafusion` | `comet` | speedup |
   | --- | --- | --- | --- | --- |
   | 2 | constant key | 1.873 ms | 41.8 us | 44.9x |
   | 8 | constant key | 6.181 ms | 110.4 us | 56.0x |
   | 32 | constant key | 18.85 ms | 376.8 us | 50.0x |
   | 2 | per-row key | 1.946 ms | 145.0 us | 13.4x |
   | 8 | per-row key | 6.390 ms | 470.7 us | 13.6x |
   | 32 | per-row key | 19.68 ms | 1.795 ms | 11.0x |
   
   **End to end**, reproducing the issue's shape (2,000,000 rows, `attrs map<string, string>`, 10%
   `NULL` maps, 0-6 entries each, 60 distinct keys, 400 distinct values; `noop` sink, median of 5
   after 2 warmups; Apple silicon, so the absolute numbers differ from the issue's Ryzen run):
   
   | query | Spark | Comet before | Comet after |
   | --- | --- | --- | --- |
   | `SELECT attrs FROM t` | 1098 ms | 716 ms | 856 ms |
   | `SELECT size(attrs) FROM t` | 291 ms | 214 ms | 210 ms |
   | `SELECT size(map_keys(attrs)) FROM t` | 289 ms | 246 ms | 214 ms |
   | `SELECT size(map_values(attrs)) FROM t` | 276 ms | 231 ms | 204 ms |
   | `SELECT element_at(attrs, 'a1') FROM t` | 506 ms | 278 ms | **211 ms** |
   | combined query from the issue | 602 ms | 307 ms | 273 ms |
   
   Taking `size(attrs)` as the control, as the issue does, the isolated cost of the lookup goes from
   **+64 ms to +1.5 ms**, and `element_at` is no longer the outlier among the map kernels. The
   Spark-arm numbers drifted about 15% between the two runs on this machine, so read the Comet columns
   against each other and against their own control rather than the ratios.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: vectorize the native map lookup behind element_at and GetMapValue [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #5806:
URL: https://github.com/apache/datafusion-comet/pull/5806#issuecomment-5605325013

   Reviewed against the PR head `4f6a1cee80`.
   
   Overall this is a well-built change. The approach is right, the Arrow usage is correct, and the two hazards specific to this kernel (sliced `MapArray` entry offsets, per-row lookup keys) are both handled and both tested at the Rust and SQL levels. No blockers from me.
   
   ## Major
   
   ### 1. The map's validity buffer is never consulted, so a NULL map row is answered from its entries
   
   **Where:** `native/spark-expr/src/map_funcs/map_extract.rs:181-192`
   
   The per-row scan derives NULL purely from "no matching entry":
   
   ```rust
   let found = (start..end).find(|&i| matched.value(i));
   if let Some(i) = found { indices[row] = (i + entries_start) as u32; }
   nulls.append(found.is_some());
   ```
   
   `map_array.nulls()` is never read. A row marked NULL in the `MapArray` whose offset window is non-empty returns that entry's value instead of NULL.
   
   Spark's `GetMapValue` and `ElementAt` are `BinaryExpression`s evaluated through `nullSafeEval`, so a NULL map operand returns NULL unconditionally, independent of what the underlying storage holds. Arrow does not require a null list/map slot to have a zero-length offset window, only that offsets stay monotonic, so the code relies on an invariant it does not own.
   
   This is *not* a regression against the pinned DataFusion 55.0.0, which has the same gap (`general_map_extract_inner` iterates `value_offsets().windows(2)` with no validity test and passes `None` as the output `ListArray`'s null buffer). But upstream closed exactly this gap in apache/datafusion#24999, which added `if map_array.is_valid(row_index)` plus `map_array.nulls().cloned()`. Since this PR forks the kernel into Comet, Comet will not inherit that fix when the DataFusion pin moves.
   
   I traced the producers that could deliver such an input (arrow-rs Parquet list reader, arrow-java `ListVector.setValueCount`, `MutableArrayData::extend_nulls` behind DataFusion's `CaseExpr` scatter, `filter`, `concat`, Comet's own `spark_map_sort`) and all of them emit degenerate offsets for null rows. So I could not construct a query that triggers it today. Latent rather than live.
   
   **Suggested direction:** one line at the end of the scan, before building `indices`:
   
   ```rust
   let nulls = NullBuffer::union(map_array.nulls(), nulls.finish().as_ref());
   ```
   
   `map_array.nulls()` is already sliced to the visible rows, so no offset arithmetic is needed. A Rust unit test building a `MapArray` via `MapArray::try_new` with a null row over a non-empty offset window is the only way to construct the input.
   
   ## Minor
   
   ### 2. A constant map argument is expanded to one copy per row before the compare
   
   **Where:** `map_extract.rs:120-123`
   
   `ColumnarValue::Scalar(scalar) => scalar.to_array_of_size(number_rows)?` materializes `number_rows x entries` map entries, and the vectorized `eq` then runs over all of them. For an 8192-row batch against a folded map literal with `E` entries that is `8192 * E` Arrow entries built and compared per batch.
   
   This is reachable: `CometLiteral` rebuilds folded `MapType` literals natively (gated on `MapKeySupport`), and `CometMapExpressionSuite` already exercises `element_at(<folded map literal>, _1)`. It is not a regression (DF 55's `make_scalar_function` expands scalars identically), and the very large case in the suite goes through `CometCreateMap`'s JVM dispatcher rather than a literal, so the exposure is bounded by how large a map literal survives as a `Literal`. Still, in a PR whose thesis is "hoist the constant out of the row loop", the constant *map* is left un-hoisted while the constant *key* is hoisted.
   
   If you take it, the cheap version is a scalar-map branch that compares the key array against the single map's `E` entry keys with `E` full-length `eq` calls. Memory drops from `O(N * E)` to `O(N)`. If you would rather not add a second path, a comment at line 122 recording that the expansion is deliberate and bounded would keep the next reader from re-deriving this.
   
   ### 3. `spark_map_extract` is exported publicly with no consumer
   
   **Where:** `native/spark-expr/src/lib.rs:64`, `map_funcs/mod.rs:19`
   
   `SparkMapExtract` is used by `comet_scalar_funcs.rs` and the bench. The free function `spark_map_extract` is only called from `invoke_with_args` and the module's own tests, both of which see it through `super::*`. Compare `spark_map_sort`, which is re-exported because `comet_scalar_funcs.rs` imports it by name. Dropping `pub` on the function and the re-export keeps the crate's public surface to the UDF.
   
   ### 4. `DataType::Null` in the map position is now an error rather than a passthrough
   
   **Where:** `map_extract.rs:104-112` and `124-129`
   
   DF 55's `return_type`, `coerce_types`, and `map_extract_inner` all special-case `map_type.is_null()`. The replacement does not, so a `Null`-typed first argument produces `map_extract: the first argument must be a map, got Null`. I believe this is unreachable from Comet (Spark's `ExtractValue.apply` requires a `MapType` child, and `CometElementAt.getSupportLevel` returns `Unsupported` for anything that is not an array or a map), so I would not add the branch. Worth being deliberate about it, since the doc comment claims the kernel "is never less capable than the one it replaces" and this is the one place it is.
   
   ### 5. Test coverage does not span the key types the new compare primitive accepts
   
   The compare changed from `dyn Array` / `ArrayData` equality to `arrow::compute::kernels::cmp::eq`, which is type-strict in ways `ArrayData` equality is not (decimal precision and scale, timestamp time zone strings, integer width). Everything that survives `MapKeySupport` reaches the new `eq`, but the tests only exercise `Utf8`, `Int32`, and `Binary` keys (`element_at_map.sql` covers string, int with coercion, and binary; the Rust tests cover `Utf8` and `Int32`).
   
   Not exercised anywhere: `decimal`, `date`, `timestamp` / `timestamp_ntz`, `boolean`, and the narrower integer widths as map key types. Adding a handful of rows to `element_at_map.sql` is cheap relative to the risk that one of these takes the `exec_err!` path in `key_match_mask` at runtime instead of matching.
   
   ## Questions
   
   ### 6. The override is registered by name only, so `element_at` still resolves to DataFusion's implementation
   
   DataFusion's `SessionState::register_udf` registers a UDF under its aliases as well as its name, and upstream `MapExtract` declares `aliases: ["element_at"]`. `SparkMapExtract` declares none, so after `register_all_comet_functions` the registry holds `map_extract -> SparkMapExtract` and `element_at -> MapExtract` (list-returning). Nothing in Comet emits the name `element_at` (grepped `spark/src/main`, `native/core/src`, `native/spark-expr/src`), so this is inert today, and it was equally inert before the PR because the deleted `planner.rs` arm also matched only `"map_extract"`. Worth either adding the alias or noting in the doc comment that the override is name-scoped, so a future `element_at` emission does not silently get a one-element list?
   
   ### 7. The PR description overstates one of the behavior changes
   
   > a lookup key whose runtime type is not the map's key type is now rejected rather than silently missing every row
   
   DF 55's `map_extract_inner` already does this:
   
   ```rust
   if key_type != key_arg.data_type() {
       return exec_err!("The key type {} does not match the map key type {}", ...);
   }
   ```
   
   So that part is a no-op change, only the message text differs. Worth correcting in the description so it is not weighed as new risk.
   
   There *is* a real null-semantics improvement here that the description does not claim. DF 55 compares a NULL lookup key against a NULL stored key with `ArrayData` equality, which reports them equal, so a map carrying a NULL key (reachable through the known `map_from_arrays` gap, #4680) would return that key's value where Spark returns NULL. The new kernel returns NULL for a NULL lookup key on both the scalar path (line 150-153) and the array path (the mask is intersected with the comparison's null buffer at line 228). Upstream's `make_comparator` fix does not close this, since `SortOptions::default()` also treats two nulls as equal. That is a genuine correctness win worth mentioning in the description.
   
   ## Scope
   
   Appropriately scoped. Kernel, registration, the planner cleanup the kernel enables, a bench, and tests. No drive-by refactoring, no unrelated formatting, no new configuration, no API change beyond the one noted in finding 3.
   
   The `elementwise_match_mask` backstop (lines 239-253) is the only arguably speculative piece, since `MapKeySupport` declines every key type `eq` rejects, so it is unreachable from Comet. It faithfully reproduces DF 55's comparison, is 15 lines, and is unit-tested, so I would keep it. `datafusion-comet-spark-expr` is a published crate with consumers outside Comet's serde gate.
   
   No duplicated abstraction. `spark_map_sort` handles the sliced-`MapArray` trap by rebasing offsets rather than windowing them, which is right for its output shape, so there is no helper to extract.
   
   ## Spark compatibility
   
   Checked against `complexTypeExtractors.scala`. `GetMapValue.nullSafeEval` delegates to `GetMapValueUtil.getValueEval`, which scans entries in order and stops at the first `ordering.equiv` hit, returning null when not found or when the matched value is null. `ElementAt`'s map overload shares that path. Both are `nullSafeEval`, so a NULL map or a NULL key short-circuits to NULL. Since SPARK-40066 (Spark 3.4) neither throws under ANSI for a missing key, so there is no ANSI divergence to reproduce here.
   
   Matches Spark: first-match-wins, missing key, NULL key, NULL map with a degenerate window, empty map, NULL stored value. Improves on the replaced kernel for a NULL lookup key against a NULL stored key.
   
   Diverges: the NULL map row with a non-empty window (finding 1). Pre-existing, not introduced here.
   
   Remaining risk is carried entirely by `MapKeySupport`, unchanged by this PR. The gate is correct for the new kernel for the same reasons it was correct for the old one, since both compare raw Arrow values rather than Spark's normalized keys.
   
   One thing worth adding to the description: removing the `ListExtract` wrapper also decouples Comet from apache/datafusion#24999, which changed the absent-key result from a one-element NULL list to an empty list. `ListExtract` with `fail_on_error=false` would have absorbed that, but not depending on it is better. That answers my earlier question on this PR.
   
   ## Tests
   
   Present and load-bearing:
   
   - Sliced `MapArray` entry offsets, at both the Rust level (`sliced_map_keeps_original_entry_offsets`, constant and per-row key) and the SQL level (`element_at on a sliced map reads the visible entries`, via `ORDER BY ... LIMIT ... OFFSET`). Reverting the `entries_start` arithmetic fails both.
   - Per-row lookup key including NULL key and NULL map, Rust and SQL.
   - Duplicate keys resolving to the first match, which pins the `find`-stops-at-first-hit contract.
   - Empty-window fast path, empty input, scalar map argument, non-string keys, the nested-key backstop, and both rejection paths.
   
   Missing:
   
   - A NULL map row over a non-empty offset window (finding 1). Only constructible in Rust.
   - Key types beyond `Utf8` / `Int32` / `Binary` (finding 5).
   - A scalar map combined with an array key. The code path exists at line 120 crossed with line 157 and is untested, though the risk is low.
   
   The two new Scala tests would pass on the pre-PR implementation too, which is correct for what they are. They are regression tests for the new kernel's specific hazards, not for a bug the PR fixes, and the sliced-map one does fail if the windowing is wrong.
   
   Non-nullable map value types (`valueContainsNull = false`) are covered indirectly by the existing `map_entries(element_at(map(1, map(1, 2)), _1))` test, which routes a non-nullable-value map out of `element_at`.
   
   ## Performance
   
   Evidence is required for this PR and it is supplied at both levels, which is the right shape. The new criterion bench runs both implementations over the same inputs so the comparison stays reproducible, and it is parameterized over entries-per-map and constant versus per-row key.
   
   The kernel-level claim (11x to 56x) is credible from the code. DF 55 allocates two `ArrayRef` slices per candidate entry and compares through `dyn Array` equality, which the new path replaces with one `eq` over the batch plus a bit scan plus one `take`. I confirmed the pinned 55.0.0 source is the slow slice-based version, so the baseline is the one Comet actually ships.
   
   By dimension:
   
   - CPU: large reduction on the constant-key path, smaller but real on the per-row path.
   - Allocations: large reduction. The remaining per-call allocations are the `gather` vector and the gathered per-entry key array on the per-row path, both `O(total entries in the batch)`, plus `indices` at `O(rows)`.
   - Memory: the per-row path materializes a full per-entry copy of the lookup key at line 174, which for string keys copies the key bytes once per entry rather than once per row. Bounded and worth the trade. The scalar-map expansion at line 122 is the unbounded one, see finding 2.
   - I/O, shuffle, latency: unaffected.
   
   No regression risk I can see for the shapes that were already fast. The `entries_start == entries_end` early return keeps the all-empty and all-NULL case at one `new_null_array`.
   
   The end-to-end table reads honestly, including the caveat that the Spark arm drifted about 15% between runs and that the Comet columns should be read against their own `size(attrs)` control. `+64 ms -> +1.5 ms` isolated lookup cost is consistent with the kernel numbers.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: vectorize the native map lookup behind element_at and GetMapValue [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5806:
URL: https://github.com/apache/datafusion-comet/pull/5806#issuecomment-5607870344

   The null-map validity concern is fixed in 7d527c148, and your counterexample was exact.
   
   The gather validity only recorded whether a match existed, so it never consulted the map's own null buffer. The kernel now checks it before the mask scan and skips a null row's entries entirely, which is also marginally cheaper than scanning them and discarding the result.
   
   I built your shape and confirmed it reads the wrong value before the fix rather than trusting the reading. `offsets [0, 1, 3]`, validity `[false, true]`, physical entries `a -> 7`, `a -> 1`, `b -> 2`, so row 0 is NULL but still spans a live `a -> 7`:
   
   ```
   null_map_row_reads_null_even_with_live_entries ... FAILED
     left: [Some(7), Some(1)]
    right: [None, Some(1)]
   null_map_row_reads_null_with_a_per_row_lookup_key ... FAILED
     left: [Some(7), Some(2)]
    right: [None, Some(2)]
   ```
   
   Both pass with the fix, and all 24 `map_funcs` tests pass. `cargo clippy -p datafusion-comet-spark-expr --all-targets -- -D warnings` and `cargo fmt --all -- --check` are clean.
   
   I covered both key paths deliberately, because they fail for slightly different reasons. The constant-key path compares the null row's entry against the broadcast key. The per-row path gathers a key per entry, so the null row's entry is compared against that row's own lookup key, and the mask is set there too. Masking at the gather-index step covers both, which is why I put it there rather than in `key_match_mask`.
   
   On why there is no Scala test: this shape is not reachable from Spark SQL. Spark's own map builders give a null row an empty offset range, so an end-to-end `NULL` map already read `NULL` before this change, and the existing suite covers that. What produces a null row over retained entries is a null mask added above intact child buffers inside Comet, which is why the guard belongs at the kernel boundary where every caller gets the same contract, and why the regression has to be built at that boundary too. If you would rather I also asserted the reachable case end to end, the existing `constant_key_hit_and_miss` row 3 and the Scala suite's whole-map-null cases already do.
   
   I left the wider `map_extract` gap in pinned DataFusion 55 alone, as you noted it is pre-existing there. Worth an upstream issue, but not something this PR should carry.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: vectorize the native map lookup behind element_at and GetMapValue [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5806:
URL: https://github.com/apache/datafusion-comet/pull/5806#discussion_r3981712381


##########
spark/src/test/scala/org/apache/comet/CometMapExpressionSuite.scala:
##########
@@ -460,6 +460,34 @@ class CometMapExpressionSuite extends CometTestBase {
     }
   }
 
+  // The native lookup compares a whole batch of map entries in one pass and then reads each row's
+  // window out of the resulting mask. A native OFFSET slices the batch, and Arrow keeps a sliced
+  // MapArray's original entry offsets, so the visible entries start part way into the keys child --
+  // the same trap `mapsort` hit below. Reading the mask from index 0 would answer every row with
+  // some other row's entries.
+  test("element_at on a sliced map reads the visible entries") {

Review Comment:
   You were right that it was slicing by accident. It looks up through `_2['a3']` / `_2['shared']` now, and the fixture carries NULL map rows so the distinction is load-bearing rather than latent; I kept one `element_at` column for that serde path. I checked that it still detects the bug rather than assuming: replacing `keys.slice(entries_start, window_len)` with `slice(0, window_len)` fails the test on the answer, and it passes again with the arithmetic restored.



##########
native/spark-expr/src/map_funcs/map_extract.rs:
##########
@@ -0,0 +1,632 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+    new_null_array, Array, ArrayRef, BooleanBufferBuilder, MapArray, NullBufferBuilder, Scalar,
+    UInt32Array,
+};
+use arrow::buffer::BooleanBuffer;
+use arrow::compute::kernels::cmp::eq;
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef};
+use datafusion::common::utils::take_function_args;
+use datafusion::common::{exec_err, Result as DataFusionResult};
+use datafusion::logical_expr::{
+    ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, Volatility,
+};
+use std::sync::Arc;
+
+/// Spark's map lookup: `GetMapValue` (`m[k]`) and `element_at(<map>, k)`.
+///
+/// Overrides DataFusion's `map_extract` under the same name, and differs from it in two ways:
+///
+///   - it returns the matched **value** rather than a one-element list, so the planner does not
+///     have to unwrap the list with a second `ListExtract` pass (see `planner.rs`);
+///   - the lookup is vectorized. DataFusion's `general_map_extract_inner` re-slices the query key
+///     and every candidate key into a fresh `ArrayRef` per comparison and compares them through
+///     `dyn Array` equality, which made a constant-key lookup roughly 35x more expensive than any
+///     other Comet map kernel and slower than Spark itself
+///     ([#5795](https://github.com/apache/datafusion-comet/issues/5795)). Here a single Arrow
+///     `eq` covers the whole batch of entries at once, the per-row work is a bit scan over the
+///     resulting mask, and the values are gathered with one `take`.
+///
+/// Spark's own lookup returns the first entry whose key compares equal, so the mask scan stops at
+/// the first match too. A missing key, a `NULL` map row, and a `NULL` lookup key all produce
+/// `NULL`, matching `GetMapValueUtil.getValueEval` and `ElementAt`'s map overload.
+///
+/// Key types whose Spark equality this cannot reproduce (floating point, non-default collations,
+/// complex keys) never reach here: `MapKeySupport` in `serde/maps.scala` declines them so the
+/// expression falls back to Spark.
+#[derive(Debug, Hash, Eq, PartialEq)]
+pub struct SparkMapExtract {
+    signature: Signature,
+}
+
+impl Default for SparkMapExtract {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl SparkMapExtract {
+    pub fn new() -> Self {
+        Self {
+            // `user_defined` so `coerce_types` runs and casts the lookup key to the map's key
+            // type; Comet's planner applies that coercion to the argument expression.
+            signature: Signature::user_defined(Volatility::Immutable),
+        }
+    }
+}
+
+impl ScalarUDFImpl for SparkMapExtract {
+    fn name(&self) -> &str {
+        "map_extract"
+    }
+
+    fn signature(&self) -> &Signature {
+        &self.signature
+    }
+
+    fn return_type(&self, arg_types: &[DataType]) -> DataFusionResult<DataType> {
+        let [map_type, _] = take_function_args(self.name(), arg_types)?;
+        Ok(map_entry_fields(map_type)?.1.data_type().clone())
+    }
+
+    fn coerce_types(&self, arg_types: &[DataType]) -> DataFusionResult<Vec<DataType>> {
+        let [map_type, _] = take_function_args(self.name(), arg_types)?;
+        Ok(vec![
+            map_type.clone(),
+            map_entry_fields(map_type)?.0.data_type().clone(),
+        ])
+    }
+
+    fn invoke_with_args(&self, args: ScalarFunctionArgs) -> DataFusionResult<ColumnarValue> {
+        let [map_arg, key_arg] = take_function_args(self.name(), &args.args)?;
+        spark_map_extract(map_arg, key_arg, args.number_rows)
+    }
+}
+
+/// The `(key, value)` fields of a `Map`'s entry struct.
+fn map_entry_fields(map_type: &DataType) -> DataFusionResult<(&FieldRef, &FieldRef)> {
+    match map_type {
+        DataType::Map(entries, _) => match entries.data_type() {
+            DataType::Struct(fields) if fields.len() == 2 => Ok((&fields[0], &fields[1])),
+            other => exec_err!("map_extract: map entries must be a two-field struct, got {other}"),
+        },
+        other => exec_err!("map_extract: the first argument must be a map, got {other}"),
+    }
+}
+
+/// Look up `key_arg` in each row of `map_arg`, returning the matched value or `NULL`.
+pub fn spark_map_extract(
+    map_arg: &ColumnarValue,
+    key_arg: &ColumnarValue,
+    number_rows: usize,
+) -> DataFusionResult<ColumnarValue> {
+    let map_ref: ArrayRef = match map_arg {
+        ColumnarValue::Array(array) => Arc::clone(array),
+        ColumnarValue::Scalar(scalar) => scalar.to_array_of_size(number_rows)?,
+    };
+    let Some(map_array) = map_ref.as_any().downcast_ref::<MapArray>() else {
+        return exec_err!(
+            "map_extract: the first argument must be a map, got {}",
+            map_ref.data_type()
+        );
+    };
+
+    let num_rows = map_array.len();
+    let value_type = map_array.value_type();
+
+    // Arrow keeps a sliced `MapArray`'s entries child intact and slices only the offsets, so the
+    // offsets index the *unsliced* keys/values and the visible entries are the half-open range
+    // [entries_start, entries_end). Comparing only that window keeps a native OFFSET from paying
+    // for the entries it skipped.
+    let offsets = map_array.offsets();
+    let entries_start = offsets[0] as usize;
+    let entries_end = offsets[num_rows] as usize;
+    if entries_start == entries_end {
+        // Every row is empty or NULL, so nothing can match.
+        return Ok(ColumnarValue::Array(new_null_array(value_type, num_rows)));
+    }
+    let window_len = entries_end - entries_start;
+    let keys = map_array.keys().slice(entries_start, window_len);
+
+    let matched = match key_arg {
+        ColumnarValue::Scalar(scalar) => {
+            if scalar.is_null() {
+                // Spark map keys are never NULL, so a NULL lookup key matches nothing.
+                return Ok(ColumnarValue::Array(new_null_array(value_type, num_rows)));
+            }
+            let key = scalar.to_array_of_size(1)?;
+            key_match_mask(&keys, &key, true)?
+        }
+        ColumnarValue::Array(key_array) => {
+            if key_array.len() != num_rows {
+                return exec_err!(
+                    "map_extract: expected {num_rows} lookup keys, got {}",
+                    key_array.len()
+                );
+            }
+            // One vectorized compare needs a lookup key per *entry*, not per row, so gather each
+            // row's key across that row's entries. Entries in a gap between two rows (offsets are
+            // only required to be monotonic) keep index 0; the per-row scan below never reads
+            // those positions.
+            let mut gather = vec![0u32; window_len];
+            for row in 0..num_rows {
+                let start = offsets[row] as usize - entries_start;
+                let end = offsets[row + 1] as usize - entries_start;
+                gather[start..end].fill(row as u32);
+            }
+            let per_entry_key = take(key_array, &UInt32Array::from(gather), None)?;
+            key_match_mask(&keys, &per_entry_key, false)?
+        }
+    };
+
+    // Gather the first matching entry of each row. Map offsets are `i32`, so an entry index always
+    // fits in `u32`.
+    //
+    // A NULL map row reads NULL whatever its entries hold. Arrow does not require a null row's
+    // offset range to be empty, and neither Comet's struct-field helper (which adds a parent null
+    // mask while preserving the child buffers) nor the UDF execution layer clears those entries, so
+    // a null row can carry a live `a -> 7` that would otherwise match. Spark returns NULL for a
+    // NULL map under both ANSI modes, for `element_at` and for `GetMapValue` alike, and only
+    // `element_at` has a nullable-input guard upstream of this kernel.
+    let map_nulls = map_array.nulls();
+    let mut indices = vec![0u32; num_rows];
+    let mut nulls = NullBufferBuilder::new(num_rows);
+    for row in 0..num_rows {
+        if map_nulls.is_some_and(|n| n.is_null(row)) {
+            nulls.append(false);
+            continue;
+        }
+        let start = offsets[row] as usize - entries_start;
+        let end = offsets[row + 1] as usize - entries_start;
+        let found = (start..end).find(|&i| matched.value(i));
+        if let Some(i) = found {
+            indices[row] = (i + entries_start) as u32;
+        }
+        nulls.append(found.is_some());
+    }
+    let indices = UInt32Array::new(indices.into(), nulls.finish());
+
+    Ok(ColumnarValue::Array(take(
+        map_array.values(),
+        &indices,
+        None,
+    )?))
+}
+
+/// A bit per map entry: set where the stored key equals the lookup key. `lookup` is either a
+/// length-1 array broadcast over every entry (constant key) or one key per entry.
+fn key_match_mask(
+    keys: &ArrayRef,
+    lookup: &ArrayRef,
+    lookup_is_scalar: bool,
+) -> DataFusionResult<BooleanBuffer> {
+    // The planner casts the lookup key to the map's declared key type, so a mismatch here means
+    // the runtime encoding is not the declared one (a dictionary-encoded key column, say). Reject
+    // it rather than comparing incomparable encodings and reporting every row as a miss.
+    if keys.data_type() != lookup.data_type() {
+        return exec_err!(
+            "map_extract: lookup key type {} does not match the map key type {}",
+            lookup.data_type(),
+            keys.data_type()
+        );
+    }
+    let compared = if lookup_is_scalar {
+        eq(keys, &Scalar::new(Arc::clone(lookup)))
+    } else {
+        eq(keys, lookup)
+    };
+    match compared {
+        Ok(mask) => {
+            // A NULL on either side compares as NULL, which is not a match.
+            let (values, nulls) = mask.into_parts();
+            Ok(match nulls {
+                Some(nulls) if nulls.null_count() > 0 => &values & nulls.inner(),
+                _ => values,
+            })
+        }
+        // `eq` rejects nested key types. `MapKeySupport` declines those before they reach the
+        // native lookup, but keep DataFusion's element-wise comparison as a backstop so this
+        // kernel is never less capable than the one it replaces.
+        Err(_) => Ok(elementwise_match_mask(keys, lookup, lookup_is_scalar)),
+    }
+}
+
+fn elementwise_match_mask(
+    keys: &ArrayRef,
+    lookup: &ArrayRef,
+    lookup_is_scalar: bool,
+) -> BooleanBuffer {
+    let mut builder = BooleanBufferBuilder::new(keys.len());
+    for i in 0..keys.len() {
+        let lookup_row = if lookup_is_scalar { 0 } else { i };
+        builder.append(
+            !lookup.is_null(lookup_row)
+                && keys.slice(i, 1).as_ref() == lookup.slice(lookup_row, 1).as_ref(),
+        );
+    }
+    builder.finish()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use arrow::array::{Int32Array, StringArray, StructArray};
+    use arrow::buffer::{NullBuffer, OffsetBuffer};
+    use arrow::datatypes::{Field, Fields};
+    use datafusion::common::ScalarValue;
+
+    /// One row of a test map: `None` for a NULL map, otherwise its `(key, value)` entries.
+    type MapRow<'a> = Option<Vec<(&'a str, Option<i32>)>>;
+
+    /// `{"a": 1, "b": 2}`, `{}`, `{"c": 3, "a": 30}`, NULL, `{"b": NULL}`
+    fn test_map() -> MapArray {
+        map_from(vec![
+            Some(vec![("a", Some(1)), ("b", Some(2))]),
+            Some(vec![]),
+            Some(vec![("c", Some(3)), ("a", Some(30))]),
+            None,
+            Some(vec![("b", None)]),
+        ])
+    }
+
+    fn map_from(rows: Vec<MapRow>) -> MapArray {
+        let mut keys = Vec::new();
+        let mut values = Vec::new();
+        let mut offsets = vec![0i32];
+        let mut nulls = NullBufferBuilder::new(rows.len());
+        for row in &rows {
+            match row {
+                Some(entries) => {
+                    for (k, v) in entries {
+                        keys.push(*k);
+                        values.push(*v);
+                    }
+                    nulls.append(true);
+                }
+                None => nulls.append(false),
+            }
+            offsets.push(keys.len() as i32);
+        }
+
+        let key_field = Arc::new(Field::new("key", DataType::Utf8, false));
+        let value_field = Arc::new(Field::new("value", DataType::Int32, true));
+        let entries = StructArray::new(
+            Fields::from(vec![Arc::clone(&key_field), Arc::clone(&value_field)]),
+            vec![
+                Arc::new(StringArray::from(keys)) as ArrayRef,
+                Arc::new(Int32Array::from(values)) as ArrayRef,
+            ],
+            None,
+        );
+        let entries_field = Arc::new(Field::new(
+            "entries",
+            DataType::Struct(Fields::from(vec![key_field, value_field])),
+            false,
+        ));
+        MapArray::try_new(
+            entries_field,
+            OffsetBuffer::new(offsets.into()),
+            entries,
+            nulls.finish(),
+            false,
+        )
+        .unwrap()
+    }
+
+    fn extract(map: MapArray, key: ColumnarValue) -> Vec<Option<i32>> {
+        let num_rows = map.len();
+        let result = spark_map_extract(&ColumnarValue::Array(Arc::new(map)), &key, num_rows)
+            .unwrap()
+            .into_array(num_rows)
+            .unwrap();
+        let result = result.as_any().downcast_ref::<Int32Array>().unwrap();
+        (0..result.len())
+            .map(|i| (!result.is_null(i)).then(|| result.value(i)))
+            .collect()
+    }
+
+    fn key(value: &str) -> ColumnarValue {
+        ColumnarValue::Scalar(ScalarValue::Utf8(Some(value.to_string())))
+    }
+
+    #[test]
+    fn constant_key_hit_and_miss() {
+        // A found key returns its value; an empty row, a NULL row, and a row without the key all
+        // return NULL, as does a row whose stored value is NULL.
+        assert_eq!(
+            extract(test_map(), key("a")),
+            vec![Some(1), None, Some(30), None, None]
+        );
+        assert_eq!(
+            extract(test_map(), key("b")),
+            vec![Some(2), None, None, None, None]
+        );
+        assert_eq!(extract(test_map(), key("zz")), vec![None; 5]);
+    }
+
+    #[test]
+    fn duplicate_keys_return_the_first_match() {
+        // Spark's `GetMapValueUtil` scans entries in order and stops at the first equal key, so a
+        // map that kept duplicates (EXCEPTION dedup is a write-side check) resolves to the first.
+        let map = map_from(vec![Some(vec![("a", Some(1)), ("a", Some(2))])]);
+        assert_eq!(extract(map, key("a")), vec![Some(1)]);
+    }
+
+    #[test]
+    fn null_lookup_key_matches_nothing() {
+        assert_eq!(
+            extract(test_map(), ColumnarValue::Scalar(ScalarValue::Utf8(None))),
+            vec![None; 5]
+        );
+    }
+
+    /// A NULL row whose entries were retained rather than dropped. `map_from` gives a NULL row an
+    /// empty offset range, which is the shape Arrow's builders produce, but nothing in the format
+    /// requires it: adding a parent null mask over intact child buffers leaves the entries in
+    /// place. Such a row must still read NULL, not the value its live entry holds.
+    fn null_row_with_retained_entries() -> MapArray {

Review Comment:
   Done. One `map_of(keys, values, offsets, nulls)` builds all four now, so each fixture is just its own key/value types, offsets and null mask.



##########
native/spark-expr/benches/map_extract.rs:
##########
@@ -0,0 +1,131 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Benchmarks for the map lookup behind `GetMapValue` and `element_at(<map>, key)`.
+//!
+//! Each shape is run against both Comet's `SparkMapExtract` and the
+//! `datafusion-functions-nested` `map_extract` it overrides, so the gap that motivated

Review Comment:
   Added. The module doc now says to read the ratio as a measurement of the pinned 55.0.0 rather than a permanent gap, names the `make_comparator` rewrite on DataFusion main as the reason the baseline arm will get much faster at the next bump, and notes that what survives it is the rest of the case for the kernel: one `eq` plus one `take`, and the removed `ListExtract` pass.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: vectorize the native map lookup behind element_at and GetMapValue [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #5806:
URL: https://github.com/apache/datafusion-comet/pull/5806#issuecomment-5608988020

   Lint is stuck would be fixed in #5782 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: vectorize the native map lookup behind element_at and GetMapValue [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #5806:
URL: https://github.com/apache/datafusion-comet/pull/5806


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: vectorize the native map lookup behind element_at and GetMapValue [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5806:
URL: https://github.com/apache/datafusion-comet/pull/5806#issuecomment-5622658789

   Thanks @comphead — all eight taken in 68735f070, replies inline. Three of the earlier round's items are in there too.
   
   On the constant map argument (finding 2 in your first pass) I went with the comment, but the reachability turned out to be stronger than "deliberate and bounded": the scalar branch is not reachable from Comet at all. The native `Literal` proto carries no map, so `CometLiteral` expands a folded `MapType` literal into a `CreateMap` tree, and `CometCreateMap` is a `CometCodegenDispatch` — it hands the whole thing to the JVM dispatcher, which yields an array. That is what `element_at(<folded map literal>, _1)` in the suite actually exercises. The one map-producing literal shape that stays native is `MapFromArrays` over two empty arrays, which has no entries and stops at the empty-window fast path. So a second kernel path would be unexercisable by any Comet query, and I would rather not carry one; the comment records that instead.
   
   Key type coverage (finding 5): `element_at_map.sql` now covers boolean, tinyint, smallint, bigint, decimal, date, timestamp and timestamp_ntz, all as real Parquet map columns, and all of them run natively. One shape is not expressible — a narrower decimal lookup key fails Spark's analysis with `MAP_FUNCTION_DIFF_TYPES`, since `findTightestCommonType` does not widen decimals, so Spark hands the kernel a `Decimal128(10, 2)` on both sides by construction. I noted that in the fixture so nobody re-tries it.
   
   Description corrected on both counts. The key-type rejection claim is gone, and the two things that *are* new are stated instead: a NULL lookup key no longer matches a NULL stored key (your finding 7, which is the one real semantic win here), and a `DataType::Null` map argument is now an error rather than a passthrough (your finding 4 — unreachable from Comet, but it is the one place this is less capable than what it replaces, so the doc comment no longer claims otherwise). I also added your note about the `ListExtract` removal decoupling us from apache/datafusion#24999.
   
   Local on this revision: 17 `map_extract` Rust tests, `CometMapExpressionSuite` 25, `CometArrayExpressionSuite` 64, `CometSqlFileTestSuite` 474, `CometExpressionSuite` 141, all green. `cargo clippy --workspace --all-targets -- -D warnings` and `cargo fmt --all --check` clean.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: vectorize the native map lookup behind element_at and GetMapValue [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5806:
URL: https://github.com/apache/datafusion-comet/pull/5806#discussion_r3981709406


##########
native/spark-expr/src/map_funcs/map_extract.rs:
##########
@@ -0,0 +1,632 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+    new_null_array, Array, ArrayRef, BooleanBufferBuilder, MapArray, NullBufferBuilder, Scalar,
+    UInt32Array,
+};
+use arrow::buffer::BooleanBuffer;
+use arrow::compute::kernels::cmp::eq;
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef};
+use datafusion::common::utils::take_function_args;
+use datafusion::common::{exec_err, Result as DataFusionResult};
+use datafusion::logical_expr::{
+    ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, Volatility,
+};
+use std::sync::Arc;
+
+/// Spark's map lookup: `GetMapValue` (`m[k]`) and `element_at(<map>, k)`.
+///
+/// Overrides DataFusion's `map_extract` under the same name, and differs from it in two ways:
+///
+///   - it returns the matched **value** rather than a one-element list, so the planner does not
+///     have to unwrap the list with a second `ListExtract` pass (see `planner.rs`);
+///   - the lookup is vectorized. DataFusion's `general_map_extract_inner` re-slices the query key
+///     and every candidate key into a fresh `ArrayRef` per comparison and compares them through
+///     `dyn Array` equality, which made a constant-key lookup roughly 35x more expensive than any
+///     other Comet map kernel and slower than Spark itself
+///     ([#5795](https://github.com/apache/datafusion-comet/issues/5795)). Here a single Arrow
+///     `eq` covers the whole batch of entries at once, the per-row work is a bit scan over the
+///     resulting mask, and the values are gathered with one `take`.
+///
+/// Spark's own lookup returns the first entry whose key compares equal, so the mask scan stops at
+/// the first match too. A missing key, a `NULL` map row, and a `NULL` lookup key all produce
+/// `NULL`, matching `GetMapValueUtil.getValueEval` and `ElementAt`'s map overload.
+///
+/// Key types whose Spark equality this cannot reproduce (floating point, non-default collations,
+/// complex keys) never reach here: `MapKeySupport` in `serde/maps.scala` declines them so the
+/// expression falls back to Spark.
+#[derive(Debug, Hash, Eq, PartialEq)]
+pub struct SparkMapExtract {
+    signature: Signature,
+}
+
+impl Default for SparkMapExtract {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl SparkMapExtract {
+    pub fn new() -> Self {
+        Self {
+            // `user_defined` so `coerce_types` runs and casts the lookup key to the map's key
+            // type; Comet's planner applies that coercion to the argument expression.
+            signature: Signature::user_defined(Volatility::Immutable),
+        }
+    }
+}
+
+impl ScalarUDFImpl for SparkMapExtract {
+    fn name(&self) -> &str {
+        "map_extract"
+    }
+
+    fn signature(&self) -> &Signature {
+        &self.signature
+    }
+
+    fn return_type(&self, arg_types: &[DataType]) -> DataFusionResult<DataType> {
+        let [map_type, _] = take_function_args(self.name(), arg_types)?;
+        Ok(map_entry_fields(map_type)?.1.data_type().clone())
+    }
+
+    fn coerce_types(&self, arg_types: &[DataType]) -> DataFusionResult<Vec<DataType>> {
+        let [map_type, _] = take_function_args(self.name(), arg_types)?;
+        Ok(vec![
+            map_type.clone(),
+            map_entry_fields(map_type)?.0.data_type().clone(),
+        ])
+    }
+
+    fn invoke_with_args(&self, args: ScalarFunctionArgs) -> DataFusionResult<ColumnarValue> {
+        let [map_arg, key_arg] = take_function_args(self.name(), &args.args)?;
+        spark_map_extract(map_arg, key_arg, args.number_rows)
+    }
+}
+
+/// The `(key, value)` fields of a `Map`'s entry struct.
+fn map_entry_fields(map_type: &DataType) -> DataFusionResult<(&FieldRef, &FieldRef)> {
+    match map_type {
+        DataType::Map(entries, _) => match entries.data_type() {
+            DataType::Struct(fields) if fields.len() == 2 => Ok((&fields[0], &fields[1])),
+            other => exec_err!("map_extract: map entries must be a two-field struct, got {other}"),
+        },
+        other => exec_err!("map_extract: the first argument must be a map, got {other}"),
+    }
+}
+
+/// Look up `key_arg` in each row of `map_arg`, returning the matched value or `NULL`.
+pub fn spark_map_extract(
+    map_arg: &ColumnarValue,
+    key_arg: &ColumnarValue,
+    number_rows: usize,
+) -> DataFusionResult<ColumnarValue> {
+    let map_ref: ArrayRef = match map_arg {
+        ColumnarValue::Array(array) => Arc::clone(array),
+        ColumnarValue::Scalar(scalar) => scalar.to_array_of_size(number_rows)?,
+    };
+    let Some(map_array) = map_ref.as_any().downcast_ref::<MapArray>() else {
+        return exec_err!(
+            "map_extract: the first argument must be a map, got {}",
+            map_ref.data_type()
+        );
+    };
+
+    let num_rows = map_array.len();
+    let value_type = map_array.value_type();
+
+    // Arrow keeps a sliced `MapArray`'s entries child intact and slices only the offsets, so the
+    // offsets index the *unsliced* keys/values and the visible entries are the half-open range
+    // [entries_start, entries_end). Comparing only that window keeps a native OFFSET from paying
+    // for the entries it skipped.
+    let offsets = map_array.offsets();
+    let entries_start = offsets[0] as usize;
+    let entries_end = offsets[num_rows] as usize;
+    if entries_start == entries_end {

Review Comment:
   Fixed. Both checks moved into a `validate_lookup_key` helper that runs before the empty-window return, so the lookup key type and, for an array key, its length are now checked on every batch regardless of what the maps hold. `argument_checks_do_not_depend_on_the_data` pins it with an all-empty/all-NULL map: it now errors on a mismatched key type and on a short key array where it previously answered NULLs.



##########
native/spark-expr/src/map_funcs/map_extract.rs:
##########
@@ -0,0 +1,632 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+    new_null_array, Array, ArrayRef, BooleanBufferBuilder, MapArray, NullBufferBuilder, Scalar,
+    UInt32Array,
+};
+use arrow::buffer::BooleanBuffer;
+use arrow::compute::kernels::cmp::eq;
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef};
+use datafusion::common::utils::take_function_args;
+use datafusion::common::{exec_err, Result as DataFusionResult};
+use datafusion::logical_expr::{
+    ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, Volatility,
+};
+use std::sync::Arc;
+
+/// Spark's map lookup: `GetMapValue` (`m[k]`) and `element_at(<map>, k)`.
+///
+/// Overrides DataFusion's `map_extract` under the same name, and differs from it in two ways:
+///
+///   - it returns the matched **value** rather than a one-element list, so the planner does not
+///     have to unwrap the list with a second `ListExtract` pass (see `planner.rs`);
+///   - the lookup is vectorized. DataFusion's `general_map_extract_inner` re-slices the query key
+///     and every candidate key into a fresh `ArrayRef` per comparison and compares them through
+///     `dyn Array` equality, which made a constant-key lookup roughly 35x more expensive than any
+///     other Comet map kernel and slower than Spark itself
+///     ([#5795](https://github.com/apache/datafusion-comet/issues/5795)). Here a single Arrow
+///     `eq` covers the whole batch of entries at once, the per-row work is a bit scan over the
+///     resulting mask, and the values are gathered with one `take`.
+///
+/// Spark's own lookup returns the first entry whose key compares equal, so the mask scan stops at
+/// the first match too. A missing key, a `NULL` map row, and a `NULL` lookup key all produce
+/// `NULL`, matching `GetMapValueUtil.getValueEval` and `ElementAt`'s map overload.
+///
+/// Key types whose Spark equality this cannot reproduce (floating point, non-default collations,
+/// complex keys) never reach here: `MapKeySupport` in `serde/maps.scala` declines them so the
+/// expression falls back to Spark.
+#[derive(Debug, Hash, Eq, PartialEq)]
+pub struct SparkMapExtract {
+    signature: Signature,
+}
+
+impl Default for SparkMapExtract {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl SparkMapExtract {
+    pub fn new() -> Self {
+        Self {
+            // `user_defined` so `coerce_types` runs and casts the lookup key to the map's key
+            // type; Comet's planner applies that coercion to the argument expression.
+            signature: Signature::user_defined(Volatility::Immutable),
+        }
+    }
+}
+
+impl ScalarUDFImpl for SparkMapExtract {
+    fn name(&self) -> &str {
+        "map_extract"
+    }
+
+    fn signature(&self) -> &Signature {
+        &self.signature
+    }
+
+    fn return_type(&self, arg_types: &[DataType]) -> DataFusionResult<DataType> {
+        let [map_type, _] = take_function_args(self.name(), arg_types)?;
+        Ok(map_entry_fields(map_type)?.1.data_type().clone())
+    }
+
+    fn coerce_types(&self, arg_types: &[DataType]) -> DataFusionResult<Vec<DataType>> {
+        let [map_type, _] = take_function_args(self.name(), arg_types)?;
+        Ok(vec![
+            map_type.clone(),
+            map_entry_fields(map_type)?.0.data_type().clone(),
+        ])
+    }
+
+    fn invoke_with_args(&self, args: ScalarFunctionArgs) -> DataFusionResult<ColumnarValue> {
+        let [map_arg, key_arg] = take_function_args(self.name(), &args.args)?;
+        spark_map_extract(map_arg, key_arg, args.number_rows)
+    }
+}
+
+/// The `(key, value)` fields of a `Map`'s entry struct.
+fn map_entry_fields(map_type: &DataType) -> DataFusionResult<(&FieldRef, &FieldRef)> {
+    match map_type {
+        DataType::Map(entries, _) => match entries.data_type() {
+            DataType::Struct(fields) if fields.len() == 2 => Ok((&fields[0], &fields[1])),
+            other => exec_err!("map_extract: map entries must be a two-field struct, got {other}"),
+        },
+        other => exec_err!("map_extract: the first argument must be a map, got {other}"),
+    }
+}
+
+/// Look up `key_arg` in each row of `map_arg`, returning the matched value or `NULL`.
+pub fn spark_map_extract(
+    map_arg: &ColumnarValue,
+    key_arg: &ColumnarValue,
+    number_rows: usize,
+) -> DataFusionResult<ColumnarValue> {
+    let map_ref: ArrayRef = match map_arg {
+        ColumnarValue::Array(array) => Arc::clone(array),
+        ColumnarValue::Scalar(scalar) => scalar.to_array_of_size(number_rows)?,
+    };
+    let Some(map_array) = map_ref.as_any().downcast_ref::<MapArray>() else {
+        return exec_err!(
+            "map_extract: the first argument must be a map, got {}",
+            map_ref.data_type()
+        );
+    };
+
+    let num_rows = map_array.len();
+    let value_type = map_array.value_type();
+
+    // Arrow keeps a sliced `MapArray`'s entries child intact and slices only the offsets, so the
+    // offsets index the *unsliced* keys/values and the visible entries are the half-open range
+    // [entries_start, entries_end). Comparing only that window keeps a native OFFSET from paying
+    // for the entries it skipped.
+    let offsets = map_array.offsets();
+    let entries_start = offsets[0] as usize;
+    let entries_end = offsets[num_rows] as usize;
+    if entries_start == entries_end {
+        // Every row is empty or NULL, so nothing can match.
+        return Ok(ColumnarValue::Array(new_null_array(value_type, num_rows)));
+    }
+    let window_len = entries_end - entries_start;
+    let keys = map_array.keys().slice(entries_start, window_len);
+
+    let matched = match key_arg {
+        ColumnarValue::Scalar(scalar) => {
+            if scalar.is_null() {
+                // Spark map keys are never NULL, so a NULL lookup key matches nothing.
+                return Ok(ColumnarValue::Array(new_null_array(value_type, num_rows)));
+            }
+            let key = scalar.to_array_of_size(1)?;
+            key_match_mask(&keys, &key, true)?
+        }
+        ColumnarValue::Array(key_array) => {
+            if key_array.len() != num_rows {
+                return exec_err!(
+                    "map_extract: expected {num_rows} lookup keys, got {}",
+                    key_array.len()
+                );
+            }
+            // One vectorized compare needs a lookup key per *entry*, not per row, so gather each
+            // row's key across that row's entries. Entries in a gap between two rows (offsets are
+            // only required to be monotonic) keep index 0; the per-row scan below never reads
+            // those positions.
+            let mut gather = vec![0u32; window_len];
+            for row in 0..num_rows {
+                let start = offsets[row] as usize - entries_start;
+                let end = offsets[row + 1] as usize - entries_start;
+                gather[start..end].fill(row as u32);
+            }
+            let per_entry_key = take(key_array, &UInt32Array::from(gather), None)?;
+            key_match_mask(&keys, &per_entry_key, false)?
+        }
+    };
+
+    // Gather the first matching entry of each row. Map offsets are `i32`, so an entry index always
+    // fits in `u32`.
+    //
+    // A NULL map row reads NULL whatever its entries hold. Arrow does not require a null row's
+    // offset range to be empty, and neither Comet's struct-field helper (which adds a parent null
+    // mask while preserving the child buffers) nor the UDF execution layer clears those entries, so
+    // a null row can carry a live `a -> 7` that would otherwise match. Spark returns NULL for a
+    // NULL map under both ANSI modes, for `element_at` and for `GetMapValue` alike, and only
+    // `element_at` has a nullable-input guard upstream of this kernel.
+    let map_nulls = map_array.nulls();
+    let mut indices = vec![0u32; num_rows];
+    let mut nulls = NullBufferBuilder::new(num_rows);
+    for row in 0..num_rows {
+        if map_nulls.is_some_and(|n| n.is_null(row)) {
+            nulls.append(false);
+            continue;
+        }
+        let start = offsets[row] as usize - entries_start;
+        let end = offsets[row + 1] as usize - entries_start;
+        let found = (start..end).find(|&i| matched.value(i));
+        if let Some(i) = found {
+            indices[row] = (i + entries_start) as u32;
+        }
+        nulls.append(found.is_some());
+    }
+    let indices = UInt32Array::new(indices.into(), nulls.finish());
+
+    Ok(ColumnarValue::Array(take(
+        map_array.values(),
+        &indices,
+        None,
+    )?))
+}
+
+/// A bit per map entry: set where the stored key equals the lookup key. `lookup` is either a
+/// length-1 array broadcast over every entry (constant key) or one key per entry.
+fn key_match_mask(
+    keys: &ArrayRef,
+    lookup: &ArrayRef,
+    lookup_is_scalar: bool,
+) -> DataFusionResult<BooleanBuffer> {
+    // The planner casts the lookup key to the map's declared key type, so a mismatch here means
+    // the runtime encoding is not the declared one (a dictionary-encoded key column, say). Reject
+    // it rather than comparing incomparable encodings and reporting every row as a miss.
+    if keys.data_type() != lookup.data_type() {
+        return exec_err!(
+            "map_extract: lookup key type {} does not match the map key type {}",
+            lookup.data_type(),
+            keys.data_type()
+        );
+    }
+    let compared = if lookup_is_scalar {
+        eq(keys, &Scalar::new(Arc::clone(lookup)))
+    } else {
+        eq(keys, lookup)
+    };
+    match compared {
+        Ok(mask) => {
+            // A NULL on either side compares as NULL, which is not a match.
+            let (values, nulls) = mask.into_parts();
+            Ok(match nulls {
+                Some(nulls) if nulls.null_count() > 0 => &values & nulls.inner(),
+                _ => values,
+            })
+        }
+        // `eq` rejects nested key types. `MapKeySupport` declines those before they reach the
+        // native lookup, but keep DataFusion's element-wise comparison as a backstop so this
+        // kernel is never less capable than the one it replaces.
+        Err(_) => Ok(elementwise_match_mask(keys, lookup, lookup_is_scalar)),

Review Comment:
   Taken. It dispatches on `keys.data_type().is_nested()` now, which is exactly the predicate `compare_op` applies after unwrapping one dictionary level, and with the length and type checked up front nesting was the only remaining thing `eq` rejects. Both `eq` calls use `?`, so a genuine failure surfaces instead of silently taking the per-row path.



##########
native/spark-expr/src/map_funcs/mod.rs:
##########
@@ -15,5 +15,7 @@
 // specific language governing permissions and limitations
 // under the License.
 
+mod map_extract;
 mod map_sort;
+pub use map_extract::{spark_map_extract, SparkMapExtract};

Review Comment:
   Done. `spark_map_extract` is private again; only `SparkMapExtract` is re-exported from `map_funcs` and `lib.rs`.



##########
native/spark-expr/src/comet_scalar_funcs.rs:
##########
@@ -321,6 +321,9 @@ fn all_scalar_functions() -> Vec<Arc<ScalarUDF>> {
         )),
         Arc::new(ScalarUDF::new_from_impl(SparkMakeDate::default())),
         Arc::new(ScalarUDF::new_from_impl(SparkMakeTime::default())),
+        // Overrides datafusion-functions-nested' `map_extract` with a vectorized lookup that
+        // returns the value itself rather than a one-element list (#5795).
+        Arc::new(ScalarUDF::new_from_impl(SparkMapExtract::default())),

Review Comment:
   Added the alias rather than documenting the gap, since it is three lines and leaves nothing to rediscover. `SparkMapExtract::aliases()` returns `["element_at"]`, so the override now replaces both registry entries. I re-grepped and nothing emits that name today, so this is consistency rather than a fix, and there is a one-line test so it does not quietly come undone.



##########
native/spark-expr/src/map_funcs/map_extract.rs:
##########
@@ -0,0 +1,632 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+    new_null_array, Array, ArrayRef, BooleanBufferBuilder, MapArray, NullBufferBuilder, Scalar,
+    UInt32Array,
+};
+use arrow::buffer::BooleanBuffer;
+use arrow::compute::kernels::cmp::eq;
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef};
+use datafusion::common::utils::take_function_args;
+use datafusion::common::{exec_err, Result as DataFusionResult};
+use datafusion::logical_expr::{
+    ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, Volatility,
+};
+use std::sync::Arc;
+
+/// Spark's map lookup: `GetMapValue` (`m[k]`) and `element_at(<map>, k)`.
+///
+/// Overrides DataFusion's `map_extract` under the same name, and differs from it in two ways:
+///
+///   - it returns the matched **value** rather than a one-element list, so the planner does not
+///     have to unwrap the list with a second `ListExtract` pass (see `planner.rs`);
+///   - the lookup is vectorized. DataFusion's `general_map_extract_inner` re-slices the query key
+///     and every candidate key into a fresh `ArrayRef` per comparison and compares them through
+///     `dyn Array` equality, which made a constant-key lookup roughly 35x more expensive than any
+///     other Comet map kernel and slower than Spark itself
+///     ([#5795](https://github.com/apache/datafusion-comet/issues/5795)). Here a single Arrow
+///     `eq` covers the whole batch of entries at once, the per-row work is a bit scan over the
+///     resulting mask, and the values are gathered with one `take`.
+///
+/// Spark's own lookup returns the first entry whose key compares equal, so the mask scan stops at
+/// the first match too. A missing key, a `NULL` map row, and a `NULL` lookup key all produce
+/// `NULL`, matching `GetMapValueUtil.getValueEval` and `ElementAt`'s map overload.
+///
+/// Key types whose Spark equality this cannot reproduce (floating point, non-default collations,
+/// complex keys) never reach here: `MapKeySupport` in `serde/maps.scala` declines them so the
+/// expression falls back to Spark.
+#[derive(Debug, Hash, Eq, PartialEq)]
+pub struct SparkMapExtract {
+    signature: Signature,
+}
+
+impl Default for SparkMapExtract {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl SparkMapExtract {
+    pub fn new() -> Self {
+        Self {
+            // `user_defined` so `coerce_types` runs and casts the lookup key to the map's key
+            // type; Comet's planner applies that coercion to the argument expression.
+            signature: Signature::user_defined(Volatility::Immutable),
+        }
+    }
+}
+
+impl ScalarUDFImpl for SparkMapExtract {
+    fn name(&self) -> &str {
+        "map_extract"
+    }
+
+    fn signature(&self) -> &Signature {
+        &self.signature
+    }
+
+    fn return_type(&self, arg_types: &[DataType]) -> DataFusionResult<DataType> {
+        let [map_type, _] = take_function_args(self.name(), arg_types)?;
+        Ok(map_entry_fields(map_type)?.1.data_type().clone())
+    }
+
+    fn coerce_types(&self, arg_types: &[DataType]) -> DataFusionResult<Vec<DataType>> {
+        let [map_type, _] = take_function_args(self.name(), arg_types)?;
+        Ok(vec![
+            map_type.clone(),
+            map_entry_fields(map_type)?.0.data_type().clone(),
+        ])
+    }
+
+    fn invoke_with_args(&self, args: ScalarFunctionArgs) -> DataFusionResult<ColumnarValue> {
+        let [map_arg, key_arg] = take_function_args(self.name(), &args.args)?;
+        spark_map_extract(map_arg, key_arg, args.number_rows)
+    }
+}
+
+/// The `(key, value)` fields of a `Map`'s entry struct.
+fn map_entry_fields(map_type: &DataType) -> DataFusionResult<(&FieldRef, &FieldRef)> {
+    match map_type {
+        DataType::Map(entries, _) => match entries.data_type() {
+            DataType::Struct(fields) if fields.len() == 2 => Ok((&fields[0], &fields[1])),
+            other => exec_err!("map_extract: map entries must be a two-field struct, got {other}"),
+        },
+        other => exec_err!("map_extract: the first argument must be a map, got {other}"),
+    }
+}
+
+/// Look up `key_arg` in each row of `map_arg`, returning the matched value or `NULL`.
+pub fn spark_map_extract(
+    map_arg: &ColumnarValue,
+    key_arg: &ColumnarValue,
+    number_rows: usize,
+) -> DataFusionResult<ColumnarValue> {
+    let map_ref: ArrayRef = match map_arg {
+        ColumnarValue::Array(array) => Arc::clone(array),
+        ColumnarValue::Scalar(scalar) => scalar.to_array_of_size(number_rows)?,
+    };
+    let Some(map_array) = map_ref.as_any().downcast_ref::<MapArray>() else {
+        return exec_err!(
+            "map_extract: the first argument must be a map, got {}",
+            map_ref.data_type()
+        );
+    };
+
+    let num_rows = map_array.len();
+    let value_type = map_array.value_type();
+
+    // Arrow keeps a sliced `MapArray`'s entries child intact and slices only the offsets, so the
+    // offsets index the *unsliced* keys/values and the visible entries are the half-open range
+    // [entries_start, entries_end). Comparing only that window keeps a native OFFSET from paying
+    // for the entries it skipped.
+    let offsets = map_array.offsets();
+    let entries_start = offsets[0] as usize;
+    let entries_end = offsets[num_rows] as usize;
+    if entries_start == entries_end {
+        // Every row is empty or NULL, so nothing can match.
+        return Ok(ColumnarValue::Array(new_null_array(value_type, num_rows)));
+    }
+    let window_len = entries_end - entries_start;
+    let keys = map_array.keys().slice(entries_start, window_len);
+
+    let matched = match key_arg {
+        ColumnarValue::Scalar(scalar) => {
+            if scalar.is_null() {
+                // Spark map keys are never NULL, so a NULL lookup key matches nothing.
+                return Ok(ColumnarValue::Array(new_null_array(value_type, num_rows)));
+            }
+            let key = scalar.to_array_of_size(1)?;
+            key_match_mask(&keys, &key, true)?
+        }
+        ColumnarValue::Array(key_array) => {
+            if key_array.len() != num_rows {
+                return exec_err!(
+                    "map_extract: expected {num_rows} lookup keys, got {}",
+                    key_array.len()
+                );
+            }
+            // One vectorized compare needs a lookup key per *entry*, not per row, so gather each
+            // row's key across that row's entries. Entries in a gap between two rows (offsets are
+            // only required to be monotonic) keep index 0; the per-row scan below never reads
+            // those positions.
+            let mut gather = vec![0u32; window_len];
+            for row in 0..num_rows {
+                let start = offsets[row] as usize - entries_start;
+                let end = offsets[row + 1] as usize - entries_start;
+                gather[start..end].fill(row as u32);
+            }
+            let per_entry_key = take(key_array, &UInt32Array::from(gather), None)?;
+            key_match_mask(&keys, &per_entry_key, false)?
+        }
+    };
+
+    // Gather the first matching entry of each row. Map offsets are `i32`, so an entry index always
+    // fits in `u32`.
+    //
+    // A NULL map row reads NULL whatever its entries hold. Arrow does not require a null row's
+    // offset range to be empty, and neither Comet's struct-field helper (which adds a parent null
+    // mask while preserving the child buffers) nor the UDF execution layer clears those entries, so
+    // a null row can carry a live `a -> 7` that would otherwise match. Spark returns NULL for a
+    // NULL map under both ANSI modes, for `element_at` and for `GetMapValue` alike, and only
+    // `element_at` has a nullable-input guard upstream of this kernel.
+    let map_nulls = map_array.nulls();

Review Comment:
   I could not find a producing path either, so I softened the wording rather than claim one. The comment now says every producer traced -- the Parquet readers, Spark`s `ArrowWriter`, and arrow`s own `filter` / `take` / `concat` -- gives a null row an empty range, and that this is a representation the format permits rather than one known to arrive here, so the guard exists to pin the contract at the kernel boundary. The test fixture`s doc comment had the same phrasing and got the same treatment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: vectorize the native map lookup behind element_at and GetMapValue [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #5806:
URL: https://github.com/apache/datafusion-comet/pull/5806#discussion_r3981087399


##########
native/spark-expr/src/comet_scalar_funcs.rs:
##########
@@ -321,6 +321,9 @@ fn all_scalar_functions() -> Vec<Arc<ScalarUDF>> {
         )),
         Arc::new(ScalarUDF::new_from_impl(SparkMakeDate::default())),
         Arc::new(ScalarUDF::new_from_impl(SparkMakeTime::default())),
+        // Overrides datafusion-functions-nested' `map_extract` with a vectorized lookup that
+        // returns the value itself rather than a one-element list (#5795).
+        Arc::new(ScalarUDF::new_from_impl(SparkMapExtract::default())),

Review Comment:
   DF's `MapExtract` declares `aliases: ["element_at"]`, and `SessionState::register_udf` inserts one entry per alias. After this override, `udf("map_extract")` returns the Comet kernel but `udf("element_at")` still returns DF's list-returning one.
   
   Nothing serializes that name today, so it is latent rather than live, but the registry is now inconsistent and the next `element_at` serde would silently get the wrong shape. Either add `aliases()` returning `["element_at"]`, or note here why the alias is deliberately left alone.



##########
native/spark-expr/src/map_funcs/map_extract.rs:
##########
@@ -0,0 +1,632 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+    new_null_array, Array, ArrayRef, BooleanBufferBuilder, MapArray, NullBufferBuilder, Scalar,
+    UInt32Array,
+};
+use arrow::buffer::BooleanBuffer;
+use arrow::compute::kernels::cmp::eq;
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef};
+use datafusion::common::utils::take_function_args;
+use datafusion::common::{exec_err, Result as DataFusionResult};
+use datafusion::logical_expr::{
+    ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, Volatility,
+};
+use std::sync::Arc;
+
+/// Spark's map lookup: `GetMapValue` (`m[k]`) and `element_at(<map>, k)`.
+///
+/// Overrides DataFusion's `map_extract` under the same name, and differs from it in two ways:
+///
+///   - it returns the matched **value** rather than a one-element list, so the planner does not
+///     have to unwrap the list with a second `ListExtract` pass (see `planner.rs`);
+///   - the lookup is vectorized. DataFusion's `general_map_extract_inner` re-slices the query key
+///     and every candidate key into a fresh `ArrayRef` per comparison and compares them through
+///     `dyn Array` equality, which made a constant-key lookup roughly 35x more expensive than any
+///     other Comet map kernel and slower than Spark itself
+///     ([#5795](https://github.com/apache/datafusion-comet/issues/5795)). Here a single Arrow
+///     `eq` covers the whole batch of entries at once, the per-row work is a bit scan over the
+///     resulting mask, and the values are gathered with one `take`.
+///
+/// Spark's own lookup returns the first entry whose key compares equal, so the mask scan stops at
+/// the first match too. A missing key, a `NULL` map row, and a `NULL` lookup key all produce
+/// `NULL`, matching `GetMapValueUtil.getValueEval` and `ElementAt`'s map overload.
+///
+/// Key types whose Spark equality this cannot reproduce (floating point, non-default collations,
+/// complex keys) never reach here: `MapKeySupport` in `serde/maps.scala` declines them so the
+/// expression falls back to Spark.
+#[derive(Debug, Hash, Eq, PartialEq)]
+pub struct SparkMapExtract {
+    signature: Signature,
+}
+
+impl Default for SparkMapExtract {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl SparkMapExtract {
+    pub fn new() -> Self {
+        Self {
+            // `user_defined` so `coerce_types` runs and casts the lookup key to the map's key
+            // type; Comet's planner applies that coercion to the argument expression.
+            signature: Signature::user_defined(Volatility::Immutable),
+        }
+    }
+}
+
+impl ScalarUDFImpl for SparkMapExtract {
+    fn name(&self) -> &str {
+        "map_extract"
+    }
+
+    fn signature(&self) -> &Signature {
+        &self.signature
+    }
+
+    fn return_type(&self, arg_types: &[DataType]) -> DataFusionResult<DataType> {
+        let [map_type, _] = take_function_args(self.name(), arg_types)?;
+        Ok(map_entry_fields(map_type)?.1.data_type().clone())
+    }
+
+    fn coerce_types(&self, arg_types: &[DataType]) -> DataFusionResult<Vec<DataType>> {
+        let [map_type, _] = take_function_args(self.name(), arg_types)?;
+        Ok(vec![
+            map_type.clone(),
+            map_entry_fields(map_type)?.0.data_type().clone(),
+        ])
+    }
+
+    fn invoke_with_args(&self, args: ScalarFunctionArgs) -> DataFusionResult<ColumnarValue> {
+        let [map_arg, key_arg] = take_function_args(self.name(), &args.args)?;
+        spark_map_extract(map_arg, key_arg, args.number_rows)
+    }
+}
+
+/// The `(key, value)` fields of a `Map`'s entry struct.
+fn map_entry_fields(map_type: &DataType) -> DataFusionResult<(&FieldRef, &FieldRef)> {
+    match map_type {
+        DataType::Map(entries, _) => match entries.data_type() {
+            DataType::Struct(fields) if fields.len() == 2 => Ok((&fields[0], &fields[1])),
+            other => exec_err!("map_extract: map entries must be a two-field struct, got {other}"),
+        },
+        other => exec_err!("map_extract: the first argument must be a map, got {other}"),
+    }
+}
+
+/// Look up `key_arg` in each row of `map_arg`, returning the matched value or `NULL`.
+pub fn spark_map_extract(
+    map_arg: &ColumnarValue,
+    key_arg: &ColumnarValue,
+    number_rows: usize,
+) -> DataFusionResult<ColumnarValue> {
+    let map_ref: ArrayRef = match map_arg {
+        ColumnarValue::Array(array) => Arc::clone(array),
+        ColumnarValue::Scalar(scalar) => scalar.to_array_of_size(number_rows)?,
+    };
+    let Some(map_array) = map_ref.as_any().downcast_ref::<MapArray>() else {
+        return exec_err!(
+            "map_extract: the first argument must be a map, got {}",
+            map_ref.data_type()
+        );
+    };
+
+    let num_rows = map_array.len();
+    let value_type = map_array.value_type();
+
+    // Arrow keeps a sliced `MapArray`'s entries child intact and slices only the offsets, so the
+    // offsets index the *unsliced* keys/values and the visible entries are the half-open range
+    // [entries_start, entries_end). Comparing only that window keeps a native OFFSET from paying
+    // for the entries it skipped.
+    let offsets = map_array.offsets();
+    let entries_start = offsets[0] as usize;
+    let entries_end = offsets[num_rows] as usize;
+    if entries_start == entries_end {

Review Comment:
   Argument validation is now data-dependent: this early return runs before the key-length check (L158) and the key-type check (L223). DF 55.0.0's `map_extract_inner` validates the key type before touching data, so a mismatch failed identically on every batch. Now a batch whose maps are all empty or all NULL returns NULLs, while a later batch with entries errors. Same query, failure depends on which partition holds data.
   
   Both checks are cheap and neither needs the entries window, so they can be hoisted above this return.



##########
native/spark-expr/src/map_funcs/mod.rs:
##########
@@ -15,5 +15,7 @@
 // specific language governing permissions and limitations
 // under the License.
 
+mod map_extract;
 mod map_sort;
+pub use map_extract::{spark_map_extract, SparkMapExtract};

Review Comment:
   `spark_map_extract` has no caller outside this module. `comet_scalar_funcs.rs` and the new bench both import only `SparkMapExtract`. (`spark_map_sort` is exported because `comet_scalar_funcs.rs` calls it directly.)
   
   The tests are in the same file, so the free function can stay private and only the UDF needs exporting.



##########
spark/src/test/scala/org/apache/comet/CometMapExpressionSuite.scala:
##########
@@ -460,6 +460,34 @@ class CometMapExpressionSuite extends CometTestBase {
     }
   }
 
+  // The native lookup compares a whole batch of map entries in one pass and then reads each row's
+  // window out of the resulting mask. A native OFFSET slices the batch, and Arrow keeps a sliced
+  // MapArray's original entry offsets, so the visible entries start part way into the keys child --
+  // the same trap `mapsort` hit below. Reading the mask from index 0 would answer every row with
+  // some other row's entries.
+  test("element_at on a sliced map reads the visible entries") {

Review Comment:
   This does exercise `entries_start != 0` today, but incidentally.
   
   `_2` is a nullable Parquet map column, so under the Spark 4.1 ANSI default `needsNullGuard` wraps the lookup in `CASE WHEN _2 IS NOT NULL`, and DF's `CaseExpr` evaluates the THEN branch through `filter_record_batch`. It preserves the slice only because the predicate is all-true, so arrow picks `IterationStrategy::All` and returns `values.slice(0, count)`. Add one NULL map row and the filter compacts the entries, `entries_start` becomes 0, and the test quietly stops testing what its comment says.
   
   An ANSI-off variant, or `_2['a3']` (`GetMapValue` has no guard) so the sliced map always reaches the kernel, would pin the intent. This is analysis, not an observed failure.



##########
native/spark-expr/benches/map_extract.rs:
##########
@@ -0,0 +1,131 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Benchmarks for the map lookup behind `GetMapValue` and `element_at(<map>, key)`.
+//!
+//! Each shape is run against both Comet's `SparkMapExtract` and the
+//! `datafusion-functions-nested` `map_extract` it overrides, so the gap that motivated

Review Comment:
   Worth a line here: DF main has already rewritten `general_map_extract_inner` to a single `make_comparator` over the batch, so the per-comparison slicing is 55.0.0-specific. This gap will narrow a lot at the next DF bump and the comparison will start measuring something different.
   
   That does not change the case for the Comet kernel (one `eq` plus one `take`, plus the removed `ListExtract` pass), but readers should not take these ratios as permanent.



##########
native/spark-expr/src/map_funcs/map_extract.rs:
##########
@@ -0,0 +1,632 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+    new_null_array, Array, ArrayRef, BooleanBufferBuilder, MapArray, NullBufferBuilder, Scalar,
+    UInt32Array,
+};
+use arrow::buffer::BooleanBuffer;
+use arrow::compute::kernels::cmp::eq;
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef};
+use datafusion::common::utils::take_function_args;
+use datafusion::common::{exec_err, Result as DataFusionResult};
+use datafusion::logical_expr::{
+    ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, Volatility,
+};
+use std::sync::Arc;
+
+/// Spark's map lookup: `GetMapValue` (`m[k]`) and `element_at(<map>, k)`.
+///
+/// Overrides DataFusion's `map_extract` under the same name, and differs from it in two ways:
+///
+///   - it returns the matched **value** rather than a one-element list, so the planner does not
+///     have to unwrap the list with a second `ListExtract` pass (see `planner.rs`);
+///   - the lookup is vectorized. DataFusion's `general_map_extract_inner` re-slices the query key
+///     and every candidate key into a fresh `ArrayRef` per comparison and compares them through
+///     `dyn Array` equality, which made a constant-key lookup roughly 35x more expensive than any
+///     other Comet map kernel and slower than Spark itself
+///     ([#5795](https://github.com/apache/datafusion-comet/issues/5795)). Here a single Arrow
+///     `eq` covers the whole batch of entries at once, the per-row work is a bit scan over the
+///     resulting mask, and the values are gathered with one `take`.
+///
+/// Spark's own lookup returns the first entry whose key compares equal, so the mask scan stops at
+/// the first match too. A missing key, a `NULL` map row, and a `NULL` lookup key all produce
+/// `NULL`, matching `GetMapValueUtil.getValueEval` and `ElementAt`'s map overload.
+///
+/// Key types whose Spark equality this cannot reproduce (floating point, non-default collations,
+/// complex keys) never reach here: `MapKeySupport` in `serde/maps.scala` declines them so the
+/// expression falls back to Spark.
+#[derive(Debug, Hash, Eq, PartialEq)]
+pub struct SparkMapExtract {
+    signature: Signature,
+}
+
+impl Default for SparkMapExtract {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl SparkMapExtract {
+    pub fn new() -> Self {
+        Self {
+            // `user_defined` so `coerce_types` runs and casts the lookup key to the map's key
+            // type; Comet's planner applies that coercion to the argument expression.
+            signature: Signature::user_defined(Volatility::Immutable),
+        }
+    }
+}
+
+impl ScalarUDFImpl for SparkMapExtract {
+    fn name(&self) -> &str {
+        "map_extract"
+    }
+
+    fn signature(&self) -> &Signature {
+        &self.signature
+    }
+
+    fn return_type(&self, arg_types: &[DataType]) -> DataFusionResult<DataType> {
+        let [map_type, _] = take_function_args(self.name(), arg_types)?;
+        Ok(map_entry_fields(map_type)?.1.data_type().clone())
+    }
+
+    fn coerce_types(&self, arg_types: &[DataType]) -> DataFusionResult<Vec<DataType>> {
+        let [map_type, _] = take_function_args(self.name(), arg_types)?;
+        Ok(vec![
+            map_type.clone(),
+            map_entry_fields(map_type)?.0.data_type().clone(),
+        ])
+    }
+
+    fn invoke_with_args(&self, args: ScalarFunctionArgs) -> DataFusionResult<ColumnarValue> {
+        let [map_arg, key_arg] = take_function_args(self.name(), &args.args)?;
+        spark_map_extract(map_arg, key_arg, args.number_rows)
+    }
+}
+
+/// The `(key, value)` fields of a `Map`'s entry struct.
+fn map_entry_fields(map_type: &DataType) -> DataFusionResult<(&FieldRef, &FieldRef)> {
+    match map_type {
+        DataType::Map(entries, _) => match entries.data_type() {
+            DataType::Struct(fields) if fields.len() == 2 => Ok((&fields[0], &fields[1])),
+            other => exec_err!("map_extract: map entries must be a two-field struct, got {other}"),
+        },
+        other => exec_err!("map_extract: the first argument must be a map, got {other}"),
+    }
+}
+
+/// Look up `key_arg` in each row of `map_arg`, returning the matched value or `NULL`.
+pub fn spark_map_extract(
+    map_arg: &ColumnarValue,
+    key_arg: &ColumnarValue,
+    number_rows: usize,
+) -> DataFusionResult<ColumnarValue> {
+    let map_ref: ArrayRef = match map_arg {
+        ColumnarValue::Array(array) => Arc::clone(array),
+        ColumnarValue::Scalar(scalar) => scalar.to_array_of_size(number_rows)?,
+    };
+    let Some(map_array) = map_ref.as_any().downcast_ref::<MapArray>() else {
+        return exec_err!(
+            "map_extract: the first argument must be a map, got {}",
+            map_ref.data_type()
+        );
+    };
+
+    let num_rows = map_array.len();
+    let value_type = map_array.value_type();
+
+    // Arrow keeps a sliced `MapArray`'s entries child intact and slices only the offsets, so the
+    // offsets index the *unsliced* keys/values and the visible entries are the half-open range
+    // [entries_start, entries_end). Comparing only that window keeps a native OFFSET from paying
+    // for the entries it skipped.
+    let offsets = map_array.offsets();
+    let entries_start = offsets[0] as usize;
+    let entries_end = offsets[num_rows] as usize;
+    if entries_start == entries_end {
+        // Every row is empty or NULL, so nothing can match.
+        return Ok(ColumnarValue::Array(new_null_array(value_type, num_rows)));
+    }
+    let window_len = entries_end - entries_start;
+    let keys = map_array.keys().slice(entries_start, window_len);
+
+    let matched = match key_arg {
+        ColumnarValue::Scalar(scalar) => {
+            if scalar.is_null() {
+                // Spark map keys are never NULL, so a NULL lookup key matches nothing.
+                return Ok(ColumnarValue::Array(new_null_array(value_type, num_rows)));
+            }
+            let key = scalar.to_array_of_size(1)?;
+            key_match_mask(&keys, &key, true)?
+        }
+        ColumnarValue::Array(key_array) => {
+            if key_array.len() != num_rows {
+                return exec_err!(
+                    "map_extract: expected {num_rows} lookup keys, got {}",
+                    key_array.len()
+                );
+            }
+            // One vectorized compare needs a lookup key per *entry*, not per row, so gather each
+            // row's key across that row's entries. Entries in a gap between two rows (offsets are
+            // only required to be monotonic) keep index 0; the per-row scan below never reads
+            // those positions.
+            let mut gather = vec![0u32; window_len];
+            for row in 0..num_rows {
+                let start = offsets[row] as usize - entries_start;
+                let end = offsets[row + 1] as usize - entries_start;
+                gather[start..end].fill(row as u32);
+            }
+            let per_entry_key = take(key_array, &UInt32Array::from(gather), None)?;
+            key_match_mask(&keys, &per_entry_key, false)?
+        }
+    };
+
+    // Gather the first matching entry of each row. Map offsets are `i32`, so an entry index always
+    // fits in `u32`.
+    //
+    // A NULL map row reads NULL whatever its entries hold. Arrow does not require a null row's
+    // offset range to be empty, and neither Comet's struct-field helper (which adds a parent null
+    // mask while preserving the child buffers) nor the UDF execution layer clears those entries, so
+    // a null row can carry a live `a -> 7` that would otherwise match. Spark returns NULL for a
+    // NULL map under both ANSI modes, for `element_at` and for `GetMapValue` alike, and only
+    // `element_at` has a nullable-input guard upstream of this kernel.
+    let map_nulls = map_array.nulls();
+    let mut indices = vec![0u32; num_rows];
+    let mut nulls = NullBufferBuilder::new(num_rows);
+    for row in 0..num_rows {
+        if map_nulls.is_some_and(|n| n.is_null(row)) {
+            nulls.append(false);
+            continue;
+        }
+        let start = offsets[row] as usize - entries_start;
+        let end = offsets[row + 1] as usize - entries_start;
+        let found = (start..end).find(|&i| matched.value(i));
+        if let Some(i) = found {
+            indices[row] = (i + entries_start) as u32;
+        }
+        nulls.append(found.is_some());
+    }
+    let indices = UInt32Array::new(indices.into(), nulls.finish());
+
+    Ok(ColumnarValue::Array(take(
+        map_array.values(),
+        &indices,
+        None,
+    )?))
+}
+
+/// A bit per map entry: set where the stored key equals the lookup key. `lookup` is either a
+/// length-1 array broadcast over every entry (constant key) or one key per entry.
+fn key_match_mask(
+    keys: &ArrayRef,
+    lookup: &ArrayRef,
+    lookup_is_scalar: bool,
+) -> DataFusionResult<BooleanBuffer> {
+    // The planner casts the lookup key to the map's declared key type, so a mismatch here means
+    // the runtime encoding is not the declared one (a dictionary-encoded key column, say). Reject
+    // it rather than comparing incomparable encodings and reporting every row as a miss.
+    if keys.data_type() != lookup.data_type() {
+        return exec_err!(
+            "map_extract: lookup key type {} does not match the map key type {}",
+            lookup.data_type(),
+            keys.data_type()
+        );
+    }
+    let compared = if lookup_is_scalar {
+        eq(keys, &Scalar::new(Arc::clone(lookup)))
+    } else {
+        eq(keys, lookup)
+    };
+    match compared {
+        Ok(mask) => {
+            // A NULL on either side compares as NULL, which is not a match.
+            let (values, nulls) = mask.into_parts();
+            Ok(match nulls {
+                Some(nulls) if nulls.null_count() > 0 => &values & nulls.inner(),
+                _ => values,
+            })
+        }
+        // `eq` rejects nested key types. `MapKeySupport` declines those before they reach the
+        // native lookup, but keep DataFusion's element-wise comparison as a backstop so this
+        // kernel is never less capable than the one it replaces.
+        Err(_) => Ok(elementwise_match_mask(keys, lookup, lookup_is_scalar)),

Review Comment:
   This catches any `ArrowError`, not just the nested-type rejection the comment describes. A future or unrelated `eq` failure silently drops into `elementwise_match_mask`, which is exactly the per-row slice-and-compare pattern this PR exists to remove. That is a silent ~50x throughput cliff with no signal, and an untestable branch.
   
   Deciding from the type up front would keep genuine errors visible:
   
   ```rust
   if keys.data_type().is_nested() { elementwise_match_mask(..) } else { eq(..)? }
   ```



##########
native/spark-expr/src/map_funcs/map_extract.rs:
##########
@@ -0,0 +1,632 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+    new_null_array, Array, ArrayRef, BooleanBufferBuilder, MapArray, NullBufferBuilder, Scalar,
+    UInt32Array,
+};
+use arrow::buffer::BooleanBuffer;
+use arrow::compute::kernels::cmp::eq;
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef};
+use datafusion::common::utils::take_function_args;
+use datafusion::common::{exec_err, Result as DataFusionResult};
+use datafusion::logical_expr::{
+    ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, Volatility,
+};
+use std::sync::Arc;
+
+/// Spark's map lookup: `GetMapValue` (`m[k]`) and `element_at(<map>, k)`.
+///
+/// Overrides DataFusion's `map_extract` under the same name, and differs from it in two ways:
+///
+///   - it returns the matched **value** rather than a one-element list, so the planner does not
+///     have to unwrap the list with a second `ListExtract` pass (see `planner.rs`);
+///   - the lookup is vectorized. DataFusion's `general_map_extract_inner` re-slices the query key
+///     and every candidate key into a fresh `ArrayRef` per comparison and compares them through
+///     `dyn Array` equality, which made a constant-key lookup roughly 35x more expensive than any
+///     other Comet map kernel and slower than Spark itself
+///     ([#5795](https://github.com/apache/datafusion-comet/issues/5795)). Here a single Arrow
+///     `eq` covers the whole batch of entries at once, the per-row work is a bit scan over the
+///     resulting mask, and the values are gathered with one `take`.
+///
+/// Spark's own lookup returns the first entry whose key compares equal, so the mask scan stops at
+/// the first match too. A missing key, a `NULL` map row, and a `NULL` lookup key all produce
+/// `NULL`, matching `GetMapValueUtil.getValueEval` and `ElementAt`'s map overload.
+///
+/// Key types whose Spark equality this cannot reproduce (floating point, non-default collations,
+/// complex keys) never reach here: `MapKeySupport` in `serde/maps.scala` declines them so the
+/// expression falls back to Spark.
+#[derive(Debug, Hash, Eq, PartialEq)]
+pub struct SparkMapExtract {
+    signature: Signature,
+}
+
+impl Default for SparkMapExtract {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl SparkMapExtract {
+    pub fn new() -> Self {
+        Self {
+            // `user_defined` so `coerce_types` runs and casts the lookup key to the map's key
+            // type; Comet's planner applies that coercion to the argument expression.
+            signature: Signature::user_defined(Volatility::Immutable),
+        }
+    }
+}
+
+impl ScalarUDFImpl for SparkMapExtract {
+    fn name(&self) -> &str {
+        "map_extract"
+    }
+
+    fn signature(&self) -> &Signature {
+        &self.signature
+    }
+
+    fn return_type(&self, arg_types: &[DataType]) -> DataFusionResult<DataType> {
+        let [map_type, _] = take_function_args(self.name(), arg_types)?;
+        Ok(map_entry_fields(map_type)?.1.data_type().clone())
+    }
+
+    fn coerce_types(&self, arg_types: &[DataType]) -> DataFusionResult<Vec<DataType>> {
+        let [map_type, _] = take_function_args(self.name(), arg_types)?;
+        Ok(vec![
+            map_type.clone(),
+            map_entry_fields(map_type)?.0.data_type().clone(),
+        ])
+    }
+
+    fn invoke_with_args(&self, args: ScalarFunctionArgs) -> DataFusionResult<ColumnarValue> {
+        let [map_arg, key_arg] = take_function_args(self.name(), &args.args)?;
+        spark_map_extract(map_arg, key_arg, args.number_rows)
+    }
+}
+
+/// The `(key, value)` fields of a `Map`'s entry struct.
+fn map_entry_fields(map_type: &DataType) -> DataFusionResult<(&FieldRef, &FieldRef)> {
+    match map_type {
+        DataType::Map(entries, _) => match entries.data_type() {
+            DataType::Struct(fields) if fields.len() == 2 => Ok((&fields[0], &fields[1])),
+            other => exec_err!("map_extract: map entries must be a two-field struct, got {other}"),
+        },
+        other => exec_err!("map_extract: the first argument must be a map, got {other}"),
+    }
+}
+
+/// Look up `key_arg` in each row of `map_arg`, returning the matched value or `NULL`.
+pub fn spark_map_extract(
+    map_arg: &ColumnarValue,
+    key_arg: &ColumnarValue,
+    number_rows: usize,
+) -> DataFusionResult<ColumnarValue> {
+    let map_ref: ArrayRef = match map_arg {
+        ColumnarValue::Array(array) => Arc::clone(array),
+        ColumnarValue::Scalar(scalar) => scalar.to_array_of_size(number_rows)?,
+    };
+    let Some(map_array) = map_ref.as_any().downcast_ref::<MapArray>() else {
+        return exec_err!(
+            "map_extract: the first argument must be a map, got {}",
+            map_ref.data_type()
+        );
+    };
+
+    let num_rows = map_array.len();
+    let value_type = map_array.value_type();
+
+    // Arrow keeps a sliced `MapArray`'s entries child intact and slices only the offsets, so the
+    // offsets index the *unsliced* keys/values and the visible entries are the half-open range
+    // [entries_start, entries_end). Comparing only that window keeps a native OFFSET from paying
+    // for the entries it skipped.
+    let offsets = map_array.offsets();
+    let entries_start = offsets[0] as usize;
+    let entries_end = offsets[num_rows] as usize;
+    if entries_start == entries_end {
+        // Every row is empty or NULL, so nothing can match.
+        return Ok(ColumnarValue::Array(new_null_array(value_type, num_rows)));
+    }
+    let window_len = entries_end - entries_start;
+    let keys = map_array.keys().slice(entries_start, window_len);
+
+    let matched = match key_arg {
+        ColumnarValue::Scalar(scalar) => {
+            if scalar.is_null() {
+                // Spark map keys are never NULL, so a NULL lookup key matches nothing.
+                return Ok(ColumnarValue::Array(new_null_array(value_type, num_rows)));
+            }
+            let key = scalar.to_array_of_size(1)?;
+            key_match_mask(&keys, &key, true)?
+        }
+        ColumnarValue::Array(key_array) => {
+            if key_array.len() != num_rows {
+                return exec_err!(
+                    "map_extract: expected {num_rows} lookup keys, got {}",
+                    key_array.len()
+                );
+            }
+            // One vectorized compare needs a lookup key per *entry*, not per row, so gather each
+            // row's key across that row's entries. Entries in a gap between two rows (offsets are
+            // only required to be monotonic) keep index 0; the per-row scan below never reads
+            // those positions.
+            let mut gather = vec![0u32; window_len];
+            for row in 0..num_rows {
+                let start = offsets[row] as usize - entries_start;
+                let end = offsets[row + 1] as usize - entries_start;
+                gather[start..end].fill(row as u32);
+            }
+            let per_entry_key = take(key_array, &UInt32Array::from(gather), None)?;
+            key_match_mask(&keys, &per_entry_key, false)?
+        }
+    };
+
+    // Gather the first matching entry of each row. Map offsets are `i32`, so an entry index always
+    // fits in `u32`.
+    //
+    // A NULL map row reads NULL whatever its entries hold. Arrow does not require a null row's
+    // offset range to be empty, and neither Comet's struct-field helper (which adds a parent null
+    // mask while preserving the child buffers) nor the UDF execution layer clears those entries, so
+    // a null row can carry a live `a -> 7` that would otherwise match. Spark returns NULL for a
+    // NULL map under both ANSI modes, for `element_at` and for `GetMapValue` alike, and only
+    // `element_at` has a nullable-input guard upstream of this kernel.
+    let map_nulls = map_array.nulls();

Review Comment:
   Question about reachability, not about the fix.
   
   `child_with_parent_nulls` only *adds* a mask, it does not create entries, so this needs a producer that hands Comet a null struct row whose map child already spans a non-empty range. Everything I checked leaves it empty: the Parquet readers emit length 0 for a null parent, Spark's `ArrowWriter.StructWriter.setNull` recurses into children, and arrow's `filter` / `take` / `concat` compact.
   
   Keep the guard regardless. DF 55.0.0 has the same hole (`general_map_extract_inner` never consults `map_array.is_valid` and builds its `ListArray` with `None` nulls), so this closes a pre-existing latent bug cheaply, and both tests do fail without it.
   
   The ask is narrower: if you know the producing path, an end-to-end case (`SELECT s.m['a']` over a nullable `struct<m: map<..>>`) would be much stronger than a hand-built array. If you do not, consider softening "Comet's struct-field helper ... leaves the entries in place", which reads as a live path.



##########
native/spark-expr/src/map_funcs/map_extract.rs:
##########
@@ -0,0 +1,632 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::{
+    new_null_array, Array, ArrayRef, BooleanBufferBuilder, MapArray, NullBufferBuilder, Scalar,
+    UInt32Array,
+};
+use arrow::buffer::BooleanBuffer;
+use arrow::compute::kernels::cmp::eq;
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef};
+use datafusion::common::utils::take_function_args;
+use datafusion::common::{exec_err, Result as DataFusionResult};
+use datafusion::logical_expr::{
+    ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl, Signature, Volatility,
+};
+use std::sync::Arc;
+
+/// Spark's map lookup: `GetMapValue` (`m[k]`) and `element_at(<map>, k)`.
+///
+/// Overrides DataFusion's `map_extract` under the same name, and differs from it in two ways:
+///
+///   - it returns the matched **value** rather than a one-element list, so the planner does not
+///     have to unwrap the list with a second `ListExtract` pass (see `planner.rs`);
+///   - the lookup is vectorized. DataFusion's `general_map_extract_inner` re-slices the query key
+///     and every candidate key into a fresh `ArrayRef` per comparison and compares them through
+///     `dyn Array` equality, which made a constant-key lookup roughly 35x more expensive than any
+///     other Comet map kernel and slower than Spark itself
+///     ([#5795](https://github.com/apache/datafusion-comet/issues/5795)). Here a single Arrow
+///     `eq` covers the whole batch of entries at once, the per-row work is a bit scan over the
+///     resulting mask, and the values are gathered with one `take`.
+///
+/// Spark's own lookup returns the first entry whose key compares equal, so the mask scan stops at
+/// the first match too. A missing key, a `NULL` map row, and a `NULL` lookup key all produce
+/// `NULL`, matching `GetMapValueUtil.getValueEval` and `ElementAt`'s map overload.
+///
+/// Key types whose Spark equality this cannot reproduce (floating point, non-default collations,
+/// complex keys) never reach here: `MapKeySupport` in `serde/maps.scala` declines them so the
+/// expression falls back to Spark.
+#[derive(Debug, Hash, Eq, PartialEq)]
+pub struct SparkMapExtract {
+    signature: Signature,
+}
+
+impl Default for SparkMapExtract {
+    fn default() -> Self {
+        Self::new()
+    }
+}
+
+impl SparkMapExtract {
+    pub fn new() -> Self {
+        Self {
+            // `user_defined` so `coerce_types` runs and casts the lookup key to the map's key
+            // type; Comet's planner applies that coercion to the argument expression.
+            signature: Signature::user_defined(Volatility::Immutable),
+        }
+    }
+}
+
+impl ScalarUDFImpl for SparkMapExtract {
+    fn name(&self) -> &str {
+        "map_extract"
+    }
+
+    fn signature(&self) -> &Signature {
+        &self.signature
+    }
+
+    fn return_type(&self, arg_types: &[DataType]) -> DataFusionResult<DataType> {
+        let [map_type, _] = take_function_args(self.name(), arg_types)?;
+        Ok(map_entry_fields(map_type)?.1.data_type().clone())
+    }
+
+    fn coerce_types(&self, arg_types: &[DataType]) -> DataFusionResult<Vec<DataType>> {
+        let [map_type, _] = take_function_args(self.name(), arg_types)?;
+        Ok(vec![
+            map_type.clone(),
+            map_entry_fields(map_type)?.0.data_type().clone(),
+        ])
+    }
+
+    fn invoke_with_args(&self, args: ScalarFunctionArgs) -> DataFusionResult<ColumnarValue> {
+        let [map_arg, key_arg] = take_function_args(self.name(), &args.args)?;
+        spark_map_extract(map_arg, key_arg, args.number_rows)
+    }
+}
+
+/// The `(key, value)` fields of a `Map`'s entry struct.
+fn map_entry_fields(map_type: &DataType) -> DataFusionResult<(&FieldRef, &FieldRef)> {
+    match map_type {
+        DataType::Map(entries, _) => match entries.data_type() {
+            DataType::Struct(fields) if fields.len() == 2 => Ok((&fields[0], &fields[1])),
+            other => exec_err!("map_extract: map entries must be a two-field struct, got {other}"),
+        },
+        other => exec_err!("map_extract: the first argument must be a map, got {other}"),
+    }
+}
+
+/// Look up `key_arg` in each row of `map_arg`, returning the matched value or `NULL`.
+pub fn spark_map_extract(
+    map_arg: &ColumnarValue,
+    key_arg: &ColumnarValue,
+    number_rows: usize,
+) -> DataFusionResult<ColumnarValue> {
+    let map_ref: ArrayRef = match map_arg {
+        ColumnarValue::Array(array) => Arc::clone(array),
+        ColumnarValue::Scalar(scalar) => scalar.to_array_of_size(number_rows)?,
+    };
+    let Some(map_array) = map_ref.as_any().downcast_ref::<MapArray>() else {
+        return exec_err!(
+            "map_extract: the first argument must be a map, got {}",
+            map_ref.data_type()
+        );
+    };
+
+    let num_rows = map_array.len();
+    let value_type = map_array.value_type();
+
+    // Arrow keeps a sliced `MapArray`'s entries child intact and slices only the offsets, so the
+    // offsets index the *unsliced* keys/values and the visible entries are the half-open range
+    // [entries_start, entries_end). Comparing only that window keeps a native OFFSET from paying
+    // for the entries it skipped.
+    let offsets = map_array.offsets();
+    let entries_start = offsets[0] as usize;
+    let entries_end = offsets[num_rows] as usize;
+    if entries_start == entries_end {
+        // Every row is empty or NULL, so nothing can match.
+        return Ok(ColumnarValue::Array(new_null_array(value_type, num_rows)));
+    }
+    let window_len = entries_end - entries_start;
+    let keys = map_array.keys().slice(entries_start, window_len);
+
+    let matched = match key_arg {
+        ColumnarValue::Scalar(scalar) => {
+            if scalar.is_null() {
+                // Spark map keys are never NULL, so a NULL lookup key matches nothing.
+                return Ok(ColumnarValue::Array(new_null_array(value_type, num_rows)));
+            }
+            let key = scalar.to_array_of_size(1)?;
+            key_match_mask(&keys, &key, true)?
+        }
+        ColumnarValue::Array(key_array) => {
+            if key_array.len() != num_rows {
+                return exec_err!(
+                    "map_extract: expected {num_rows} lookup keys, got {}",
+                    key_array.len()
+                );
+            }
+            // One vectorized compare needs a lookup key per *entry*, not per row, so gather each
+            // row's key across that row's entries. Entries in a gap between two rows (offsets are
+            // only required to be monotonic) keep index 0; the per-row scan below never reads
+            // those positions.
+            let mut gather = vec![0u32; window_len];
+            for row in 0..num_rows {
+                let start = offsets[row] as usize - entries_start;
+                let end = offsets[row + 1] as usize - entries_start;
+                gather[start..end].fill(row as u32);
+            }
+            let per_entry_key = take(key_array, &UInt32Array::from(gather), None)?;
+            key_match_mask(&keys, &per_entry_key, false)?
+        }
+    };
+
+    // Gather the first matching entry of each row. Map offsets are `i32`, so an entry index always
+    // fits in `u32`.
+    //
+    // A NULL map row reads NULL whatever its entries hold. Arrow does not require a null row's
+    // offset range to be empty, and neither Comet's struct-field helper (which adds a parent null
+    // mask while preserving the child buffers) nor the UDF execution layer clears those entries, so
+    // a null row can carry a live `a -> 7` that would otherwise match. Spark returns NULL for a
+    // NULL map under both ANSI modes, for `element_at` and for `GetMapValue` alike, and only
+    // `element_at` has a nullable-input guard upstream of this kernel.
+    let map_nulls = map_array.nulls();
+    let mut indices = vec![0u32; num_rows];
+    let mut nulls = NullBufferBuilder::new(num_rows);
+    for row in 0..num_rows {
+        if map_nulls.is_some_and(|n| n.is_null(row)) {
+            nulls.append(false);
+            continue;
+        }
+        let start = offsets[row] as usize - entries_start;
+        let end = offsets[row + 1] as usize - entries_start;
+        let found = (start..end).find(|&i| matched.value(i));
+        if let Some(i) = found {
+            indices[row] = (i + entries_start) as u32;
+        }
+        nulls.append(found.is_some());
+    }
+    let indices = UInt32Array::new(indices.into(), nulls.finish());
+
+    Ok(ColumnarValue::Array(take(
+        map_array.values(),
+        &indices,
+        None,
+    )?))
+}
+
+/// A bit per map entry: set where the stored key equals the lookup key. `lookup` is either a
+/// length-1 array broadcast over every entry (constant key) or one key per entry.
+fn key_match_mask(
+    keys: &ArrayRef,
+    lookup: &ArrayRef,
+    lookup_is_scalar: bool,
+) -> DataFusionResult<BooleanBuffer> {
+    // The planner casts the lookup key to the map's declared key type, so a mismatch here means
+    // the runtime encoding is not the declared one (a dictionary-encoded key column, say). Reject
+    // it rather than comparing incomparable encodings and reporting every row as a miss.
+    if keys.data_type() != lookup.data_type() {
+        return exec_err!(
+            "map_extract: lookup key type {} does not match the map key type {}",
+            lookup.data_type(),
+            keys.data_type()
+        );
+    }
+    let compared = if lookup_is_scalar {
+        eq(keys, &Scalar::new(Arc::clone(lookup)))
+    } else {
+        eq(keys, lookup)
+    };
+    match compared {
+        Ok(mask) => {
+            // A NULL on either side compares as NULL, which is not a match.
+            let (values, nulls) = mask.into_parts();
+            Ok(match nulls {
+                Some(nulls) if nulls.null_count() > 0 => &values & nulls.inner(),
+                _ => values,
+            })
+        }
+        // `eq` rejects nested key types. `MapKeySupport` declines those before they reach the
+        // native lookup, but keep DataFusion's element-wise comparison as a backstop so this
+        // kernel is never less capable than the one it replaces.
+        Err(_) => Ok(elementwise_match_mask(keys, lookup, lookup_is_scalar)),
+    }
+}
+
+fn elementwise_match_mask(
+    keys: &ArrayRef,
+    lookup: &ArrayRef,
+    lookup_is_scalar: bool,
+) -> BooleanBuffer {
+    let mut builder = BooleanBufferBuilder::new(keys.len());
+    for i in 0..keys.len() {
+        let lookup_row = if lookup_is_scalar { 0 } else { i };
+        builder.append(
+            !lookup.is_null(lookup_row)
+                && keys.slice(i, 1).as_ref() == lookup.slice(lookup_row, 1).as_ref(),
+        );
+    }
+    builder.finish()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use arrow::array::{Int32Array, StringArray, StructArray};
+    use arrow::buffer::{NullBuffer, OffsetBuffer};
+    use arrow::datatypes::{Field, Fields};
+    use datafusion::common::ScalarValue;
+
+    /// One row of a test map: `None` for a NULL map, otherwise its `(key, value)` entries.
+    type MapRow<'a> = Option<Vec<(&'a str, Option<i32>)>>;
+
+    /// `{"a": 1, "b": 2}`, `{}`, `{"c": 3, "a": 30}`, NULL, `{"b": NULL}`
+    fn test_map() -> MapArray {
+        map_from(vec![
+            Some(vec![("a", Some(1)), ("b", Some(2))]),
+            Some(vec![]),
+            Some(vec![("c", Some(3)), ("a", Some(30))]),
+            None,
+            Some(vec![("b", None)]),
+        ])
+    }
+
+    fn map_from(rows: Vec<MapRow>) -> MapArray {
+        let mut keys = Vec::new();
+        let mut values = Vec::new();
+        let mut offsets = vec![0i32];
+        let mut nulls = NullBufferBuilder::new(rows.len());
+        for row in &rows {
+            match row {
+                Some(entries) => {
+                    for (k, v) in entries {
+                        keys.push(*k);
+                        values.push(*v);
+                    }
+                    nulls.append(true);
+                }
+                None => nulls.append(false),
+            }
+            offsets.push(keys.len() as i32);
+        }
+
+        let key_field = Arc::new(Field::new("key", DataType::Utf8, false));
+        let value_field = Arc::new(Field::new("value", DataType::Int32, true));
+        let entries = StructArray::new(
+            Fields::from(vec![Arc::clone(&key_field), Arc::clone(&value_field)]),
+            vec![
+                Arc::new(StringArray::from(keys)) as ArrayRef,
+                Arc::new(Int32Array::from(values)) as ArrayRef,
+            ],
+            None,
+        );
+        let entries_field = Arc::new(Field::new(
+            "entries",
+            DataType::Struct(Fields::from(vec![key_field, value_field])),
+            false,
+        ));
+        MapArray::try_new(
+            entries_field,
+            OffsetBuffer::new(offsets.into()),
+            entries,
+            nulls.finish(),
+            false,
+        )
+        .unwrap()
+    }
+
+    fn extract(map: MapArray, key: ColumnarValue) -> Vec<Option<i32>> {
+        let num_rows = map.len();
+        let result = spark_map_extract(&ColumnarValue::Array(Arc::new(map)), &key, num_rows)
+            .unwrap()
+            .into_array(num_rows)
+            .unwrap();
+        let result = result.as_any().downcast_ref::<Int32Array>().unwrap();
+        (0..result.len())
+            .map(|i| (!result.is_null(i)).then(|| result.value(i)))
+            .collect()
+    }
+
+    fn key(value: &str) -> ColumnarValue {
+        ColumnarValue::Scalar(ScalarValue::Utf8(Some(value.to_string())))
+    }
+
+    #[test]
+    fn constant_key_hit_and_miss() {
+        // A found key returns its value; an empty row, a NULL row, and a row without the key all
+        // return NULL, as does a row whose stored value is NULL.
+        assert_eq!(
+            extract(test_map(), key("a")),
+            vec![Some(1), None, Some(30), None, None]
+        );
+        assert_eq!(
+            extract(test_map(), key("b")),
+            vec![Some(2), None, None, None, None]
+        );
+        assert_eq!(extract(test_map(), key("zz")), vec![None; 5]);
+    }
+
+    #[test]
+    fn duplicate_keys_return_the_first_match() {
+        // Spark's `GetMapValueUtil` scans entries in order and stops at the first equal key, so a
+        // map that kept duplicates (EXCEPTION dedup is a write-side check) resolves to the first.
+        let map = map_from(vec![Some(vec![("a", Some(1)), ("a", Some(2))])]);
+        assert_eq!(extract(map, key("a")), vec![Some(1)]);
+    }
+
+    #[test]
+    fn null_lookup_key_matches_nothing() {
+        assert_eq!(
+            extract(test_map(), ColumnarValue::Scalar(ScalarValue::Utf8(None))),
+            vec![None; 5]
+        );
+    }
+
+    /// A NULL row whose entries were retained rather than dropped. `map_from` gives a NULL row an
+    /// empty offset range, which is the shape Arrow's builders produce, but nothing in the format
+    /// requires it: adding a parent null mask over intact child buffers leaves the entries in
+    /// place. Such a row must still read NULL, not the value its live entry holds.
+    fn null_row_with_retained_entries() -> MapArray {

Review Comment:
   Minor: the `key_field` / `value_field` / `StructArray` / `entries_field` / `try_new` scaffolding is repeated four times (here, `map_from`, `non_string_keys`, `nested_key_falls_back_to_elementwise_comparison`). One `map_of(keys, values, offsets, nulls) -> MapArray` helper would cut roughly 60 lines and make the three variants read as the variations they are.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: vectorize the native map lookup behind element_at and GetMapValue [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #5806:
URL: https://github.com/apache/datafusion-comet/pull/5806#issuecomment-5604730151

   Thanks @andygrove do you think this recent PR can also help https://github.com/apache/datafusion/pull/24999? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org