[jira] [Assigned] (SPARK-58707) JSON schema pruning returns a null corrupt record column

["Yang Jie (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yang Jie reassigned SPARK-58707:
--------------------------------

    Assignee: Yang Jie

> JSON schema pruning returns a null corrupt record column
> --------------------------------------------------------
>
>                 Key: SPARK-58707
>                 URL: https://issues.apache.org/jira/browse/SPARK-58707
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Yang Jie
>            Assignee: Yang Jie
>            Priority: Major
>              Labels: pull-request-available
>
> {{OptimizeCsvJsonExprs}} prunes the schema of a {{JsonToStructs}} down to the single field being
> selected. When that field is the corrupt record column, the pruned schema leaves the parser nothing
> to convert, so a malformed value in a dropped-away field is never detected and the corrupt record
> column comes back NULL instead of the record text.
> This is reachable with default configs: the rule requires {{options.isEmpty}}, which is the
> PERMISSIVE default, and {{spark.sql.optimizer.enableJsonExpressionOptimization}} defaults to true.
> Reproduction, schema {{a int, b int, _corrupt_record string}}, input {{ {"a": 1, "b": "bad"} }}:
> {code:scala}
> val df = Seq("""{"a": 1, "b": "bad"}""").toDS()
>   .selectExpr("from_json(value, 'a int, b int, _corrupt_record string') as p")
>   .selectExpr("p._corrupt_record")
> {code}
> ||spark.sql.optimizer.enableJsonExpressionOptimization||result||
> |true|null|
> |false|{{ {"a": 1, "b": "bad"} }}|
> The optimized plan shows the schema pruned to the corrupt column alone:
> {noformat}
> Project [from_json(StructField(_corrupt_record,StringType,true), value#1, ...)._corrupt_record ...]
> {noformat}
> Two branches are affected, {{GetStructField}} and {{CreateNamedStruct}}. The CSV branch added by
> SPARK-32968 already guards with {{schema(ordinal).name != nameOfCorruptRecord}}; the JSON branches
> do not.
> {{GetArrayStructFields}} is not affected: an {{ArrayType}} schema takes {{JsonToStructsEvaluator}}'s
> {{case other =>}}, so {{corruptFieldIndex}} is {{None}} and corrupt-record semantics do not apply
> on that path.
> The existing SPARK-33907 test misses this because its bad record is structurally malformed
> ({{ {"a" 1, ...} }}), which fails at tokenization regardless of the requested schema. Exposing the
> defect needs a type mismatch on a field that pruning drops.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org