You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/15 22:27:07 UTC

[I] get_json_object returns last value for duplicate keys, Spark returns first [datafusion-comet]

andygrove opened a new issue, #4947:
URL: https://github.com/apache/datafusion-comet/issues/4947

   ## Describe the bug
   
   `get_json_object` diverges from Spark when a JSON object contains duplicate keys.
   
   For input `{"a":1,"a":2}` and path `$.a`:
   - Comet returns `2` (the last occurrence)
   - Spark returns `1` (the first occurrence)
   
   Spark's `GetJsonObjectEvaluator.evaluatePath` (`sql/catalyst/.../json/JsonExpressionEvalUtils.scala`) stops at the first matching field once `dirty` is set, so it returns the first occurrence. Comet visits every entry and resolves to the last occurrence, matching `serde_json`'s `preserve_order` overwrite semantics rather than Spark's.
   
   This is a pre-existing divergence, not a regression. It predates the streaming rewrite in #4907 (the old `serde_json` + `preserve_order` path had the same last-wins behavior). It was noted during review of #4907.
   
   ## Steps to reproduce
   
   ```sql
   SELECT get_json_object('{"a":1,"a":2}', '$.a');
   -- Spark: 1
   -- Comet: 2
   ```
   
   ## Expected behavior
   
   Match Spark and return the first occurrence of a duplicated key.
   
   ## Additional context
   
   The fix is on the Rust side in `native/spark-expr/src/string_funcs/get_json_object.rs`: `SegmentVisitor::visit_map` should stop updating `found` once a key has matched (while still consuming the remaining entries to validate the document). The `test_duplicate_key_last_wins` test added in #4907 would need to be flipped to first-wins.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Native implementation of `get_json_object` returns last value for duplicate keys, Spark returns first [datafusion-comet]

Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on issue #4947:
URL: https://github.com/apache/datafusion-comet/issues/4947#issuecomment-5024983388

   Already marked as incompatible, downgrading the priority


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Native implementation of `get_json_object` returns last value for duplicate keys, Spark returns first [datafusion-comet]

Posted by "u70b3 (via GitHub)" <gi...@apache.org>.
u70b3 commented on issue #4947:
URL: https://github.com/apache/datafusion-comet/issues/4947#issuecomment-5004897067

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Native implementation of `get_json_object` returns last value for duplicate keys, Spark returns first [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao closed issue #4947: Native implementation of `get_json_object` returns last value for duplicate keys, Spark returns first
URL: https://github.com/apache/datafusion-comet/issues/4947


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org