[I] str_to_map does not honour mapKeyDedupPolicy=LAST_WIN [datafusion-comet]

["peterxcli (via GitHub)" <gi...@apache.org>]

peterxcli opened a new issue, #5857:
URL: https://github.com/apache/datafusion-comet/issues/5857

   ### Describe the bug
   
   With `spark.sql.mapKeyDedupPolicy=LAST_WIN`, `str_to_map` raises `[DUPLICATED_MAP_KEY]` on an input string that repeats a key, while Spark returns a map holding the last value for that key. Comet fails the query where Spark succeeds.
   
   The error also surfaces as a raw native failure rather than Spark's `SparkRuntimeException`, because nothing translates it on the JVM side.
   
   
   ### Steps to reproduce
   
   ```sql
   SET spark.sql.mapKeyDedupPolicy=LAST_WIN;
   SELECT str_to_map('a:1,b:2,a:3');
   ```
   
   Spark returns `{a -> 3, b -> 2}`. Comet raises:
   
   ```
   [DUPLICATED_MAP_KEY] Duplicate map key 'a' was found, please check the input data.
   To allow duplicate keys with last-value-wins semantics, set
   `datafusion.spark.map_key_dedup_policy` to `LAST_WIN`.
   ```
   
   The default `EXCEPTION` policy is unaffected: both engines raise on a duplicate key there, so this only shows up once the policy is changed.
   
   ### Expected behavior
   
   `str_to_map` follows `spark.sql.mapKeyDedupPolicy` the way Spark's `ArrayBasedMapBuilder` does, keeping the last value for each duplicate key under `LAST_WIN`.
   
   
   ### Additional context
   
   The native kernel already implements this. `datafusion-spark`'s `SparkStrToMap` reads `datafusion.spark.map_key_dedup_policy` (see `datafusion_common::config::SparkOptions`), which takes the same `EXCEPTION` and `LAST_WIN` values as the Spark config. Two things stop the Spark setting from reaching it:
   
   1. `spark.sql.mapKeyDedupPolicy` never crosses JNI. `CometExecIterator.serializeCometSQLConfs` sends only keys under `spark.comet.`, plus a short explicit list.
   2. `create_scalar_function_expr` in `native/core/src/execution/planner.rs` hands every `ScalarFunctionExpr` a fresh `ConfigOptions::default()`, so a kernel reading a session option sees DataFusion's defaults rather than the session's values.
   
   `CometStrToMap.getSupportLevel` does not inspect the policy either, so the expression stays `Compatible` and takes the native path under `LAST_WIN` instead of routing through the JVM codegen dispatcher.
   
   `spark/src/test/resources/sql-tests/expressions/map/str_to_map.sql` carries a matching note:
   
   ```sql
   -- TODO: Add LAST_WIN policy tests when spark.sql.mapKeyDedupPolicy config is supported
   ```
   
   #5854 fixes both points above while closing #4680, and adds `str_to_map_dedup_policy.sql` covering `LAST_WIN`. This issue records the divergence on its own, since it reaches a different expression from the one #4680 and #5589 describe and would otherwise go untracked if that PR is split up.
   
   Related: #4680 (null keys in `map_from_arrays` / `map_from_entries`), #5589 (`map_from_arrays` fallback under `LAST_WIN`).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] str_to_map does not honour mapKeyDedupPolicy=LAST_WIN [datafusion-comet]

["peterxcli (via GitHub)" <gi...@apache.org>]

peterxcli commented on issue #5857:
URL: https://github.com/apache/datafusion-comet/issues/5857#issuecomment-5632723921

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org