You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Ishaan400 (via GitHub)" <gi...@apache.org> on 2026/10/11 05:26:48 UTC

[PR] fix: coerce map lookup keys to the map key type in get_field [datafusion]

Ishaan400 opened a new pull request, #26199:
URL: https://github.com/apache/datafusion/pull/26199

   ## Which issue does this PR close?
   
   - Closes #26197.
   
   ## Rationale for this change
   
   `map[key]` returns an error when the key literal's type differs from the map's key type. The most common case is a map declared in SQL as `MAP(VARCHAR, ...)`: `VARCHAR` maps to `Utf8View`, so `m['a']` fails with `The key type Utf8 does not match the map key type Utf8View`. `m[1]` on a `MAP(INT, ...)` column fails the same way (`Int64` vs `Int32`). `map_extract(m, 'a')` and `element_at(m, 'a')` already work on the same columns.
   
   ## What changes are included in this PR?
   
   `GetFieldFunc::coerce_types` now coerces each map lookup key to the map's key type, the same way `map_extract` does. It follows the argument list through nested maps (`n['a']['aa']`) and stops at the first non-map level, because struct field names are only known from the literal values. NULL keys, nested key types, and keys that already match apart from dictionary encoding are left as they are, so those paths behave as before.
   
   Since the coerced key is a cast of a literal, the simplifier folds it into a literal of the map key type (for example `get_field(t.m, Utf8View("a"))`). Output column names are unchanged (`t.m[a]`).
   
   `strip_dictionary` in `utils.rs` is now `pub(crate)` so `get_field` can reuse it.
   
   ## What is the testing strategy for this PR?
   
   New cases at the end of `map.slt`:
   
   - `VARCHAR` keys: projection, missing key, NULL map row, filters, and the same result as `map_extract` / `element_at`
   - `INT` and `SMALLINT` keys with an `Int64` literal, including a missing key
   - nested `MAP(VARCHAR, MAP(VARCHAR, INT))` lookups, in a projection and a filter
   - an `Int32`-keyed map with struct values (`st[1]['x']`): the key is coerced, the field name is not
   - a dictionary-encoded `Utf8` key map, which already worked and still does
   
   All cases except the dictionary one fail on `main`.
   
   ## Are there any user-facing changes?
   
   Yes: `map[key]` now works when the key literal's type differs from the map key type. No public API changes.
   
   Because the key is coerced the same way as in `map_extract`, a key that cannot be cast to the map key type now fails at planning time with a cast error, as `map_extract` does (for example `m['x']` on an `INT`-keyed map, or `3000000000` on an `Int32`-keyed map). Before this change, those queries also failed, with the key-type mismatch error. A lossy cast also behaves like `map_extract`: `m[1.5]` on an `INT`-keyed map looks up key `1`. I kept this consistent with `map_extract` rather than giving `map[key]` its own rules. Happy to change it if reviewers prefer a stricter rule.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: coerce map lookup keys to the map key type in get_field [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26199:
URL: https://github.com/apache/datafusion/pull/26199#issuecomment-6406205425

   Marked as draft after 14 days without activity: merge conflicts.
   
   Please address these items and mark **Ready for review** when ready.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org