[jira] [Created] (SPARK-58707) JSON schema pruning returns a null corrupt record column

["Yang Jie (Jira)" <ji...@apache.org>]

Yang Jie created SPARK-58707:
--------------------------------

             Summary: JSON schema pruning returns a null corrupt record column
                 Key: SPARK-58707
                 URL: https://issues.apache.org/jira/browse/SPARK-58707
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Yang Jie


{{OptimizeCsvJsonExprs}} prunes the schema of a {{JsonToStructs}} down to the single field being
selected. When that field is the corrupt record column, the pruned schema leaves the parser nothing
to convert, so a malformed value in a dropped-away field is never detected and the corrupt record
column comes back NULL instead of the record text.

This is reachable with default configs: the rule requires {{options.isEmpty}}, which is the
PERMISSIVE default, and {{spark.sql.optimizer.enableJsonExpressionOptimization}} defaults to true.

Reproduction, schema {{a int, b int, _corrupt_record string}}, input {{ {"a": 1, "b": "bad"} }}:

{code:scala}
val df = Seq("""{"a": 1, "b": "bad"}""").toDS()
  .selectExpr("from_json(value, 'a int, b int, _corrupt_record string') as p")
  .selectExpr("p._corrupt_record")
{code}

||spark.sql.optimizer.enableJsonExpressionOptimization||result||
|true|null|
|false|{{ {"a": 1, "b": "bad"} }}|

The optimized plan shows the schema pruned to the corrupt column alone:

{noformat}
Project [from_json(StructField(_corrupt_record,StringType,true), value#1, ...)._corrupt_record ...]
{noformat}

Two branches are affected, {{GetStructField}} and {{CreateNamedStruct}}. The CSV branch added by
SPARK-32968 already guards with {{schema(ordinal).name != nameOfCorruptRecord}}; the JSON branches
do not.

{{GetArrayStructFields}} is not affected: an {{ArrayType}} schema takes {{JsonToStructsEvaluator}}'s
{{case other =>}}, so {{corruptFieldIndex}} is {{None}} and corrupt-record semantics do not apply
on that path.

The existing SPARK-33907 test misses this because its bad record is structurally malformed
({{ {"a" 1, ...} }}), which fails at tokenization regardless of the requested schema. Exposing the
defect needs a type mismatch on a field that pruning drops.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org