[jira] [Assigned] (SPARK-34133) [AVRO] Respect case sensitivity when performing Catalyst-to-Avro field matching

["Wenchen Fan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-34133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan reassigned SPARK-34133:
-----------------------------------

    Assignee: Erik Krogen

> [AVRO] Respect case sensitivity when performing Catalyst-to-Avro field matching
> -------------------------------------------------------------------------------
>
>                 Key: SPARK-34133
>                 URL: https://issues.apache.org/jira/browse/SPARK-34133
>             Project: Spark
>          Issue Type: Bug
>          Components: Input/Output, SQL
>    Affects Versions: 2.4.0, 3.2.0
>            Reporter: Erik Krogen
>            Assignee: Erik Krogen
>            Priority: Major
>
> Spark SQL is normally case-insensitive (by default), but currently when {{AvroSerializer}} and {{AvroDeserializer}} perform matching between Catalyst schemas and Avro schemas, the matching is done in a case-sensitive manner. So for example the following will fail:
> {code}
>       val avroSchema =
>         """
>           |{
>           |  "type" : "record",
>           |  "name" : "test_schema",
>           |  "fields" : [
>           |    {"name": "foo", "type": "int"},
>           |    {"name": "BAR", "type": "int"}
>           |  ]
>           |}
>       """.stripMargin
>       val df = Seq((1, 3), (2, 4)).toDF("FOO", "bar")
>       df.write.option("avroSchema", avroSchema).format("avro").save(savePath)
> {code}
> The same is true on the read path, if we assume {{testAvro}} has been written using the schema above, the below will fail to match the fields:
> {code}
> df.read.schema(new StructType().add("FOO", IntegerType).add("bar", IntegerType))
>   .format("avro").load(testAvro)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org