[jira] [Created] (SPARK-54595) Bind implicit schema coercion behavior with WITH SCHEMA EVOLUTION clause

["Szehon Ho (Jira)" <ji...@apache.org>]

Szehon Ho created SPARK-54595:
---------------------------------

             Summary: Bind implicit schema coercion behavior with WITH SCHEMA EVOLUTION clause
                 Key: SPARK-54595
                 URL: https://issues.apache.org/jira/browse/SPARK-54595
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 4.1.0
            Reporter: Szehon Ho


As [~aokolnychyi] tested this feature, he mentioned that as of Spark 4.1 the behavior is changed for MERGE INTO with UPDATE * , but without SCHEMA EVOLUTION clause.  

In particular:
* Source has less columns/nested fields than target => we fill with NULL or DEFAULT for inserts, and existing value for Update.  (though we disabled for nested structs by default in SPARK-54525)
* Source has more columns/fields than target => we drop the extra fields.

Initially, I thought its a good improvement of MERGE INTO, but Anton has a good point that it may be a surprise to some user.  So it may be better for now to be more conservative and keep the exact same behavior for without SCHEMA EVOLUTION clause, and relax it later once there is more clarity.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org