[jira] [Created] (SPARK-52065) Produce another plan tree with output columns (name, data type, nullability) in plan change logging

["Jungtaek Lim (Jira)" <ji...@apache.org>]

Jungtaek Lim created SPARK-52065:
------------------------------------

             Summary: Produce another plan tree with output columns (name, data type, nullability) in plan change logging
                 Key: SPARK-52065
                 URL: https://issues.apache.org/jira/browse/SPARK-52065
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 4.1.0
            Reporter: Jungtaek Lim


We recently got into very tricky issue (nullability change broke stateful operator) which required custom debug logging on plan change logging. This is because of lack of visibility for the output columns, especially their nullability, in our tree string of the plan.

To address the lack of visibility, we propose to add another tree string which focuses to produce output columns with data type and nullability.

Ideally, we shouldn't have two different tree strings and just consolidate, but in many cases, current tree string is long enough so that we had to restrict the number of fields to show, hence we think it's better to have a separate tree plan for it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org