[jira] [Created] (SPARK-39775) Regression due to AVRO-2035

["Nandini (Jira)" <ji...@apache.org>]

Nandini created SPARK-39775:
-------------------------------

             Summary: Regression due to AVRO-2035
                 Key: SPARK-39775
                 URL: https://issues.apache.org/jira/browse/SPARK-39775
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 3.2.1
            Reporter: Nandini


With the upgrade in Avro version to 1.9.0, for schema evolution Avro added https://issues.apache.org/jira/browse/AVRO-2035(enable validation of default values in schemas by default) which is causing regressions when user upgrades their Spark verion.

Repro code:
{code:java}
import org.apache.spark.sql.avro.functions._
val avroTypeStruct = s"""
                        |{
                        |  "type": "record",
                        |  "name": "struct",
                        |  "fields": [
                        |    {"name": "id", "type": "long", "default": null}
                        |  ]
                        |}""".stripMargin

val df = spark.range(10).select(struct('id).as("struct"))
{code}

Hive mitigated it by disabling this feature altogether in https://issues.apache.org/jira/browse/HIVE-24797  
Spark-Hive integration also imported the above changes in https://issues.apache.org/jira/browse/SPARK-34512

Can we have a fix for all the senarios?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org