You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2024/10/01 01:50:00 UTC
[jira] [Assigned] (SPARK-49743) OptimizeCsvJsonExpr should not change the schema of underlying StructType in GetArrayStructFields
[ https://issues.apache.org/jira/browse/SPARK-49743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan reassigned SPARK-49743:
-----------------------------------
Assignee: Nikhil Sheoran
> OptimizeCsvJsonExpr should not change the schema of underlying StructType in GetArrayStructFields
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-49743
> URL: https://issues.apache.org/jira/browse/SPARK-49743
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.2
> Reporter: Nikhil Sheoran
> Assignee: Nikhil Sheoran
> Priority: Major
> Labels: pull-request-available
>
> The `OptimizeCsvJsonExprs` rule can potentially change the schema of the underlying `StructField` if there are differences in the field used to access the struct vs the field in the underlying struct.
> This surfaces as a correctness issue where instead of picking the values for the corresponding column we end up returning NULL.
>
> A simple example query is:
> {code:java}
> SELECT
> from_json('[{"a": '||id||', "b": '|| (2*id) ||'}]', 'array<struct<a: INT, b: INT>>').a,
> from_json('[{"a": '||id||', "b": '|| (2*id) ||'}]', 'array<struct<a: INT, b: INT>>').A
> FROM
> range(3) as t{code}
>
>
> Here, the result is `[0], [1], [2]` for `a` but `[null], [null], [null]` for `A`. Since struct field accessor is case-insensitive, the result should had been `[0], [1], [2]` for both.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org