[jira] [Created] (SPARK-60070) File-format readers can produce MapData with null keys

["Qing Fu (Jira)" <ji...@apache.org>]

Qing Fu created SPARK-60070:
-------------------------------

             Summary: File-format readers can produce MapData with null keys
                 Key: SPARK-60070
                 URL: https://issues.apache.org/jira/browse/SPARK-60070
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Qing Fu


{{ArrayBasedMapBuilder}} rejects null map keys, but several readers construct {{ArrayBasedMapData}} directly without a runtime check, so a null key can reach query execution:

* {{ParquetRowConverter}} (map converter {{end()}}): the in-tree comment says the map "may contains null or duplicated map keys ... the behavior is undefined" and points to SPARK-26174, which was resolved as Incomplete in 2021 without a fix.
* {{OrcDeserializer}} and {{AvroDeserializer}} build the key array without a null check (each asserts in a comment that the format cannot produce one).
* {{HiveInspectors}} applies the {{MapObjectInspector}} key unwrapper with no null check.

SPARK-59598 makes {{GetMapValue}} / {{ElementAt}} skip null keys, but other consumers still see the malformed map: {{to_json}}, the CSV and XML generators, and {{MapZipWith}} (which treats two null keys as equal and mergeable).

Proposal for discussion: reject null keys at the reader boundary with a clear error, behind a legacy config that restores the current behavior. Open questions: the default (fail vs. drop the entry), and whether duplicate keys should be handled in the same change.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org