You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by dt...@apache.org on 2026/10/16 15:54:10 UTC

(spark) branch branch-4.x updated: [SPARK-60242][SQL] Assign CHAR/VARCHAR in from_avro and identity-check Avro map keys

This is an automated email from the ASF dual-hosted git repository.

dtenedor pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new ff82b485cce7 [SPARK-60242][SQL] Assign CHAR/VARCHAR in from_avro and identity-check Avro map keys
ff82b485cce7 is described below

commit ff82b485cce7ca6746fd9759bcd800c729d972a9
Author: Serge Rielau <se...@rielau.com>
AuthorDate: Fri Oct 16 08:53:49 2026 -0700

    [SPARK-60242][SQL] Assign CHAR/VARCHAR in from_avro and identity-check Avro map keys
    
    ### What changes were proposed in this pull request?
    
    CHAR/VARCHAR for Avro under SPARK-58794 (SPARK-60242). Behavior matches the JSON/XML matrix; this PR does not change the JSON or XML parsers.
    
    Contract:
    
    - `from_avro` assigns CHAR/VARCHAR on STRING values (pad/trim, `EXCEED_LIMIT_LENGTH`). Avro ENUM symbols are not assigned.
    - `to_avro` does not assign CHAR/VARCHAR.
    - Avro file scans do not assign STRING values (ApplyCharTypePadding still runs for CHAR columns). MAP CHAR/VARCHAR keys are identity-checked so CHAR padding cannot last-win-merge distinct keys.
    - MAP CHAR/VARCHAR keys in `from_avro` and scans use an identity length-check (`UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY`). Keys are not padded or trimmed.
    - PERMISSIVE nulls a malformed record only in `from_avro` (`AvroDataToCatalyst`). File scans do not wrap per-row PERMISSIVE around that error.
    - `FromAvro` and `AvroDataToCatalyst` persist `charVarcharStandardSemantics` and `charVarcharFirstClassTypes` at analysis so a view keeps creation-time types and map-key checks after SET.
    
    ### Why are the changes needed?
    
    Avro already stamped CHAR/VARCHAR from `spark.sql.catalyst.type` but copied UTF-8 bytes. Without assignment, `from_avro` CHAR values were unpadded; without identity MAP-key checks, file-scan CHAR keys could be padded and collide.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, when `spark.sql.charVarchar.standardSemantics.enabled` is true (first-class CHAR/VARCHAR):
    
    - `from_avro` into CHAR/VARCHAR pads/trims STRING values and rejects oversize values with `EXCEED_LIMIT_LENGTH` (the whole record is null in PERMISSIVE).
    - MAP keys of CHAR/VARCHAR that are not exactly the declared length fail with `UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY`.
    - Avro file scans with MAP CHAR/VARCHAR keys reject off-width keys rather than padding them into collisions.
    - `to_avro` does not pad or trim.
    
    This is user-facing on unreleased branches; it is not a change to a released Spark version.
    
    ### How was this patch tested?
    
    - `avro/testOnly *AvroFunctionsSuite -- -z SPARK-60242`
    - `avro/testOnly org.apache.spark.sql.avro.AvroV1Suite org.apache.spark.sql.avro.AvroV2Suite -- -z SPARK-58814`
    - `JAVA_HOME=... COURSIER_REPOSITORIES=... ./dev/lint-scala`
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Cursor Grok 4.6
    
    Closes #59468 from srielau/SPARK-60242.
    
    Authored-by: Serge Rielau <se...@rielau.com>
    Signed-off-by: Daniel Tenedorio <da...@databricks.com>
    (cherry picked from commit 6c3b628d8a978e0c0d0e54a9b7767fc62709a966)
    Signed-off-by: Daniel Tenedorio <da...@databricks.com>
---
 .../src/main/resources/error/error-conditions.json |   7 +
 .../apache/spark/sql/avro/AvroDataToCatalyst.scala |  44 +++-
 .../apache/spark/sql/avro/AvroFunctionsSuite.scala | 249 ++++++++++++++++++++-
 .../catalyst/expressions/avroSqlFunctions.scala    |  37 ++-
 .../spark/sql/catalyst/util/CharVarcharUtils.scala |  24 ++
 .../spark/sql/errors/QueryExecutionErrors.scala    |  20 +-
 .../apache/spark/sql/avro/AvroDeserializer.scala   |  82 ++++++-
 .../apache/spark/sql/CharVarcharTestSuite.scala    |  68 ++++++
 8 files changed, 514 insertions(+), 17 deletions(-)

diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json
index 8a2dd3bd44ac..f7e265f5969e 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -8783,6 +8783,13 @@
     },
     "sqlState" : "42000"
   },
+  "UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY" : {
+    "message" : [
+      "The Avro map key <key> is not a valid key for the map key type <dataType>.",
+      "A CHAR key must match the declared length exactly and a VARCHAR key must not exceed it; keys are never padded or trimmed. Use a STRING map key to accept any name."
+    ],
+    "sqlState" : "0A000"
+  },
   "UNSUPPORTED_BUILT_IN_JDBC_DIALECT" : {
     "message" : [
       "`<name>` is not a supported built-in JDBC dialect. Supported dialects are: <supportedNames>."
diff --git a/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala b/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala
index 87bb2f2a17c2..17e82804d6e3 100644
--- a/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala
+++ b/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala
@@ -25,24 +25,47 @@ import org.apache.avro.io.{BinaryDecoder, DecoderFactory}
 import org.apache.spark.SparkException
 import org.apache.spark.sql.catalyst.expressions.{ExpectsInputTypes, Expression, SpecificInternalRow, UnaryExpression}
 import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, CodeGenerator, ExprCode}
-import org.apache.spark.sql.catalyst.util.{FailFastMode, ParseMode, PermissiveMode}
+import org.apache.spark.sql.catalyst.util.{CharVarcharUtils, FailFastMode, ParseMode, PermissiveMode}
 import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.types._
 
 case class AvroDataToCatalyst(
     child: Expression,
     jsonFormatSchema: String,
-    options: Map[String, String])
+    options: Map[String, String],
+    // PERSISTED: capture at analysis (like JsonToStructs) so copy()/views keep
+    // creation-time CHAR/VARCHAR map-key checks, not the caller's session setting.
+    charVarcharStandardSemantics: Boolean,
+    charVarcharFirstClassTypes: Boolean)
   extends UnaryExpression with ExpectsInputTypes {
 
+  // Tests and 3-arg apply capture the current session flags.
+  def this(child: Expression, jsonFormatSchema: String, options: Map[String, String]) =
+    this(
+      child,
+      jsonFormatSchema,
+      options,
+      SQLConf.get.charVarcharStandardSemantics,
+      SQLConf.get.charVarcharFirstClassTypes)
+
   override def inputTypes: Seq[AbstractDataType] = Seq(BinaryType)
 
   override lazy val dataType: DataType = {
-    val dt = SchemaConverters.toSqlType(
+    val converted = SchemaConverters.toSqlType(
       expectedSchema,
       avroOptions.useStableIdForUnionType,
       avroOptions.stableIdPrefixForUnionType,
       avroOptions.recursiveFieldMaxDepth).dataType
+    // SchemaConverters restores CHAR/VARCHAR stamps from Avro even when first-class types
+    // are off. Use the analysis-time snapshot so copy() cannot drop CHAR map keys.
+    val dt = if (charVarcharFirstClassTypes) {
+      converted
+    } else {
+      // Conf-independent: replaceCharVarcharWithString would keep CHAR/VARCHAR if the
+      // live session has first-class types on.
+      CharVarcharUtils.replaceCharVarcharWithStringAlways(converted)
+    }
     parseMode match {
       // With PermissiveMode, the output Catalyst row might contain columns of null values for
       // corrupt records, even if some of the columns are not nullable in the user-provided schema.
@@ -69,7 +92,9 @@ case class AvroDataToCatalyst(
       avroOptions.datetimeRebaseModeInRead,
       avroOptions.useStableIdForUnionType,
       avroOptions.stableIdPrefixForUnionType,
-      avroOptions.recursiveFieldMaxDepth)
+      avroOptions.recursiveFieldMaxDepth,
+      applyCharVarcharAssignment = true,
+      charVarcharStandardSemanticsOverride = Some(charVarcharStandardSemantics))
 
   @transient private var decoder: BinaryDecoder = _
 
@@ -127,6 +152,9 @@ case class AvroDataToCatalyst(
 
   override def prettyName: String = "from_avro"
 
+  // Persist flags are not user arguments; omit them from EXPLAIN / Connect goldens.
+  override protected def stringArgs: Iterator[Any] = super.stringArgs.take(3)
+
   override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
     val expr = ctx.addReferenceObj("this", this)
     nullSafeCodeGen(ctx, ev, eval => {
@@ -146,3 +174,11 @@ case class AvroDataToCatalyst(
   override protected def withNewChildInternal(newChild: Expression): AvroDataToCatalyst =
     copy(child = newChild)
 }
+
+object AvroDataToCatalyst {
+  def apply(
+      child: Expression,
+      jsonFormatSchema: String,
+      options: Map[String, String]): AvroDataToCatalyst =
+    new AvroDataToCatalyst(child, jsonFormatSchema, options)
+}
diff --git a/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala b/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala
index bcc104aa3e82..b30e92c28731 100644
--- a/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala
+++ b/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala
@@ -26,15 +26,17 @@ import org.apache.avro.file.SeekableByteArrayInput
 import org.apache.avro.generic.{GenericDatumReader, GenericDatumWriter, GenericRecord, GenericRecordBuilder}
 import org.apache.avro.io.{DecoderFactory, EncoderFactory}
 
-import org.apache.spark.SparkException
+import org.apache.spark.{SparkException, SparkRuntimeException}
 import org.apache.spark.sql.{AnalysisException, Row}
 import org.apache.spark.sql.avro.{functions => Fns}
 import org.apache.spark.sql.avro.functions.{from_avro, to_avro}
 import org.apache.spark.sql.execution.LocalTableScanExec
-import org.apache.spark.sql.functions.{col, lit, struct}
+import org.apache.spark.sql.functions.{col, lit, map, struct}
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.test.SharedSparkSession
-import org.apache.spark.sql.types.{BinaryType, IntegerType, StructField, StructType}
+import org.apache.spark.sql.types.{
+  BinaryType, CharType, IntegerType, MapType, StringType, StructField, StructType, VarcharType
+}
 
 class AvroFunctionsSuite extends SharedSparkSession {
   import testImplicits._
@@ -769,6 +771,247 @@ class AvroFunctionsSuite extends SharedSparkSession {
       s"Should not throw NPE, but got: ${ex.getClass.getName}: ${ex.getMessage}")
   }
 
+  test("SPARK-60242: from_avro CHAR/VARCHAR length enforcement") {
+    withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+      val stringSchema = """{"type": "string"}"""
+      val charSchema = SchemaConverters.toAvroType(CharType(4), nullable = false).toString
+      val varcharSchema = SchemaConverters.toAvroType(VarcharType(4), nullable = false).toString
+      val stringMapSchema =
+        SchemaConverters.toAvroType(MapType(StringType, IntegerType), nullable = false).toString
+      val charMapSchema =
+        SchemaConverters.toAvroType(MapType(CharType(2), IntegerType), nullable = false).toString
+      val varcharMapSchema =
+        SchemaConverters.toAvroType(
+          MapType(VarcharType(3), IntegerType), nullable = false).toString
+
+      val padded = spark.range(1).select(
+        from_avro(to_avro(lit("ab"), stringSchema), charSchema).as("c"))
+      assert(padded.schema.head.dataType === CharType(4))
+      checkAnswer(padded.selectExpr("concat('<', c, '>')"), Row("<ab  >"))
+
+      val varcharShort = spark.range(1).select(
+        from_avro(to_avro(lit("ab"), stringSchema), varcharSchema).as("v"))
+      assert(varcharShort.schema.head.dataType === VarcharType(4))
+      checkAnswer(varcharShort, Row("ab"))
+
+      val roundTrip = spark.range(1).selectExpr("cast('ab' AS CHAR(4)) AS c")
+        .select(from_avro(to_avro($"c", charSchema), charSchema).as("c"))
+      checkAnswer(roundTrip.selectExpr("concat('<', c, '>')"), Row("<ab  >"))
+
+      def assertExceedLimit(body: => Any): Unit = {
+        val e = intercept[SparkException] { body }
+        val cause = e.getCause match {
+          case r: SparkRuntimeException => r
+          case other =>
+            Option(other).flatMap(t => Option(t.getCause)).getOrElse(other) match {
+              case r: SparkRuntimeException => r
+              case _ => fail(s"expected EXCEED_LIMIT_LENGTH cause, got: $e")
+            }
+        }
+        checkError(
+          exception = cause,
+          condition = "EXCEED_LIMIT_LENGTH",
+          parameters = Map("limit" -> "4"))
+      }
+      val oversize = spark.range(1).select(to_avro(lit("abcde"), stringSchema).as("bin"))
+      assertExceedLimit(oversize.select(from_avro($"bin", charSchema)).collect())
+      assertExceedLimit(oversize.select(from_avro($"bin", varcharSchema)).collect())
+
+      val permissive = oversize.select(
+        from_avro($"bin", charSchema, Map("mode" -> "PERMISSIVE").asJava))
+      checkAnswer(permissive, Row(null))
+
+      def assertUnsupportedAvroMapKey(
+          body: => Any, key: String, dataTypeSql: String): Unit = {
+        val error = intercept[Exception] { body }
+        val cause = Iterator.iterate[Throwable](error)(_.getCause)
+          .takeWhile(_ != null)
+          .collectFirst {
+            case e: SparkRuntimeException
+              if e.getCondition == "UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY" => e
+          }
+          .getOrElse(
+            fail("expected UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY cause", error))
+        checkError(
+          exception = cause,
+          condition = "UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY",
+          sqlState = "0A000",
+          parameters = Map("key" -> s"'$key'", "dataType" -> s""""$dataTypeSql""""))
+      }
+
+      val exactCharKey = spark.range(1).select(
+        from_avro(
+          to_avro(map(lit("ab"), lit(1)), stringMapSchema), charMapSchema).as("m"))
+      assert(exactCharKey.schema.head.dataType.asInstanceOf[MapType].keyType ===
+        CharType(2))
+      checkAnswer(exactCharKey, Row(Map("ab" -> 1)))
+
+      assertUnsupportedAvroMapKey(
+        spark.range(1).select(
+          from_avro(
+            to_avro(map(lit("a"), lit(1)), stringMapSchema), charMapSchema)).collect(),
+        "a",
+        "CHAR(2)")
+      assertUnsupportedAvroMapKey(
+        spark.range(1).select(
+          from_avro(
+            to_avro(map(lit("abc"), lit(1)), stringMapSchema), charMapSchema)).collect(),
+        "abc",
+        "CHAR(2)")
+
+      val varcharKey = spark.range(1).select(
+        from_avro(
+          to_avro(map(lit("ab"), lit(1)), stringMapSchema), varcharMapSchema).as("m"))
+      checkAnswer(varcharKey, Row(Map("ab" -> 1)))
+      val varcharSpacedKey = spark.range(1).select(
+        from_avro(
+          to_avro(map(lit("ab "), lit(1)), stringMapSchema), varcharMapSchema).as("m"))
+      checkAnswer(varcharSpacedKey, Row(Map("ab " -> 1)))
+      assertUnsupportedAvroMapKey(
+        spark.range(1).select(
+          from_avro(
+            to_avro(map(lit("abcd"), lit(1)), stringMapSchema),
+            varcharMapSchema)).collect(),
+        "abcd",
+        "VARCHAR(3)")
+
+      checkAnswer(
+        spark.range(1).select(
+          from_avro(
+            to_avro(map(lit("a"), lit(1)), stringMapSchema),
+            charMapSchema,
+            Map("mode" -> "PERMISSIVE").asJava)),
+        Row(null))
+
+      val char0MapSchema =
+        SchemaConverters.toAvroType(MapType(CharType(0), IntegerType), nullable = false)
+          .toString
+      checkAnswer(
+        spark.range(1).select(
+          from_avro(
+            to_avro(map(lit(""), lit(1)), stringMapSchema), char0MapSchema).as("m")),
+        Row(Map("" -> 1)))
+      assertUnsupportedAvroMapKey(
+        spark.range(1).select(
+          from_avro(
+            to_avro(map(lit("a"), lit(1)), stringMapSchema), char0MapSchema)).collect(),
+        "a",
+        "CHAR(0)")
+
+      val stringStructSchema =
+        SchemaConverters.toAvroType(
+          new StructType().add("c", StringType), nullable = false).toString
+      val charStructSchema =
+        SchemaConverters.toAvroType(
+          new StructType().add("c", CharType(4)), nullable = false).toString
+      checkAnswer(
+        spark.range(1).select(
+          from_avro(
+            to_avro(struct(lit("ab").as("c")), stringStructSchema),
+            charStructSchema).as("s"))
+          .selectExpr("concat('<', s.c, '>')"),
+        Row("<ab  >"))
+
+      withTempView("avro_cv_map_view") {
+        spark.range(1).select(
+          from_avro(
+            to_avro(map(lit("a"), lit(1)), stringMapSchema), charMapSchema).as("m"))
+          .createOrReplaceTempView("avro_cv_map_view")
+        withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "false") {
+          assertUnsupportedAvroMapKey(
+            spark.sql("SELECT * FROM avro_cv_map_view").collect(),
+            "a",
+            "CHAR(2)")
+        }
+      }
+
+      withTempPath { dir =>
+        val path = dir.getCanonicalPath
+        spark.range(1).select(
+          map(lit("a"), lit(1), lit("a "), lit(2)).as("m"))
+          .write.mode("overwrite").format("avro").save(path)
+        assertUnsupportedAvroMapKey(
+          spark.read.schema("m MAP<CHAR(2), INT>").format("avro").load(path).collect(),
+          "a",
+          "CHAR(2)")
+      }
+
+      withTempPath { dir =>
+        val path = dir.getCanonicalPath
+        spark.range(1).select(
+          map(lit("ab"), lit(1), lit("a "), lit(2)).as("m"))
+          .write.mode("overwrite").format("avro").save(path)
+        checkAnswer(
+          spark.read.schema("m MAP<CHAR(2), INT>").format("avro").load(path),
+          Row(Map("ab" -> 1, "a " -> 2)))
+      }
+
+      withTempPath { dir =>
+        val path = dir.getCanonicalPath
+        spark.range(1).select(map(lit("ab"), lit("xy")).as("m"))
+          .write.mode("overwrite").format("avro").save(path)
+        val nested = spark.read.schema("m MAP<CHAR(2), CHAR(4)>").format("avro").load(path)
+        checkAnswer(nested.selectExpr("concat('<', m['ab'], '>')"), Row("<xy  >"))
+      }
+
+      withTempPath { dir =>
+        val path = dir.getCanonicalPath
+        spark.range(1).select(map(lit("abcd"), lit(1)).as("m"))
+          .write.mode("overwrite").format("avro").save(path)
+        assertUnsupportedAvroMapKey(
+          spark.read.schema("m MAP<VARCHAR(3), INT>").format("avro").load(path).collect(),
+          "abcd",
+          "VARCHAR(3)")
+      }
+    }
+
+    withSQLConf(
+        SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "false",
+        SQLConf.PRESERVE_CHAR_VARCHAR_TYPE_INFO.key -> "true") {
+      val stringSchema = """{"type": "string"}"""
+      val charSchema = SchemaConverters.toAvroType(CharType(4), nullable = false).toString
+      val stringMapSchema =
+        SchemaConverters.toAvroType(MapType(StringType, IntegerType), nullable = false).toString
+      val charMapSchema =
+        SchemaConverters.toAvroType(MapType(CharType(2), IntegerType), nullable = false).toString
+      val padded = spark.range(1).select(
+        from_avro(to_avro(lit("ab"), stringSchema), charSchema).as("c"))
+      assert(padded.schema.head.dataType === CharType(4))
+      checkAnswer(padded.selectExpr("concat('<', c, '>')"), Row("<ab  >"))
+      // Off-width CHAR keys are not rejected or rewritten when standardSemantics is off.
+      // Exclude ConstantFolding so a CHAR Cast cannot pad the folded map key. Collecting
+      // MAP<CHAR(n), _> would still pad via the encoder, so to_json shows the raw key.
+      withSQLConf(SQLConf.OPTIMIZER_EXCLUDED_RULES.key ->
+          "org.apache.spark.sql.catalyst.optimizer.ConstantFolding") {
+        val shortKey = spark.range(1).select(
+          from_avro(
+            to_avro(map(lit("a"), lit(1)), stringMapSchema), charMapSchema).as("m"))
+        assert(shortKey.schema.head.dataType.asInstanceOf[MapType].keyType === CharType(2))
+        checkAnswer(shortKey.selectExpr("to_json(m)"), Row("""{"a":1}"""))
+      }
+    }
+
+    withSQLConf(
+        SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "false",
+        SQLConf.PRESERVE_CHAR_VARCHAR_TYPE_INFO.key -> "false") {
+      val stringSchema = """{"type": "string"}"""
+      val charSchema = SchemaConverters.toAvroType(CharType(4), nullable = false).toString
+      val df = spark.range(1).select(
+        from_avro(to_avro(lit("abcdef"), stringSchema), charSchema).as("c"))
+      assert(df.schema.head.dataType === StringType)
+      checkAnswer(df, Row("abcdef"))
+
+      withTempView("avro_cv_string_view") {
+        df.createOrReplaceTempView("avro_cv_string_view")
+        withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+          val fromView = spark.sql("SELECT * FROM avro_cv_string_view")
+          assert(fromView.schema.head.dataType === StringType)
+          checkAnswer(fromView, Row("abcdef"))
+        }
+      }
+    }
+  }
+
   test("SPARK-56043: from_avro with completely unparseable schema should not throw NPE") {
     val garbageSchema = "this is not valid JSON at all"
 
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala
index cb553aeb96ff..52b5ee67dd42 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala
@@ -21,6 +21,7 @@ import org.apache.spark.sql.catalyst.analysis.FunctionRegistry
 import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
 import org.apache.spark.sql.catalyst.util.ArrayBasedMapData
 import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.types.{MapType, NullType, StringType}
 import org.apache.spark.unsafe.types.UTF8String
 import org.apache.spark.util.Utils
@@ -62,12 +63,29 @@ import org.apache.spark.util.Utils
   since = "4.0.0"
 )
 // scalastyle:on line.size.limit
-case class FromAvro(child: Expression, jsonFormatSchema: Expression, options: Expression)
+case class FromAvro(
+    child: Expression,
+    jsonFormatSchema: Expression,
+    options: Expression,
+    // PERSISTED: copy()/views keep analysis-time CHAR/VARCHAR flags. FromAvro is
+    // RuntimeReplaceable, so a new replacement would otherwise re-read SQLConf.
+    // FunctionRegistry matches Expression-only ctors; keep 2-arg and 3-arg.
+    charVarcharStandardSemantics: Boolean,
+    charVarcharFirstClassTypes: Boolean)
   extends TernaryExpression with RuntimeReplaceable {
   override def first: Expression = child
   override def second: Expression = jsonFormatSchema
   override def third: Expression = options
 
+  // FunctionRegistry matches Expression-only ctors; keep 2-arg and 3-arg.
+  def this(child: Expression, jsonFormatSchema: Expression, options: Expression) =
+    this(
+      child,
+      jsonFormatSchema,
+      options,
+      SQLConf.get.charVarcharStandardSemantics,
+      SQLConf.get.charVarcharFirstClassTypes)
+
   def this(child: Expression, jsonFormatSchema: Expression) =
     this(child, jsonFormatSchema, Literal.create(null))
 
@@ -120,17 +138,30 @@ case class FromAvro(child: Expression, jsonFormatSchema: Expression, options: Ex
         Map.empty
     }
     val constructor = try {
-      Utils.classForName("org.apache.spark.sql.avro.AvroDataToCatalyst").getConstructors().head
+      // 5-arg primary: child, schema, options, plus the two persisted CHAR/VARCHAR flags.
+      Utils.classForName("org.apache.spark.sql.avro.AvroDataToCatalyst").getConstructors
+        .find { c =>
+          val t = c.getParameterTypes
+          t.length == 5 && t(3) == classOf[Boolean] && t(4) == classOf[Boolean]
+        }.get
     } catch {
       case _: java.lang.ClassNotFoundException =>
         throw QueryCompilationErrors.avroNotLoadedSqlFunctionsUnusable(functionName = "FROM_AVRO")
     }
-    val expr = constructor.newInstance(child, schemaValue, optionsValue)
+    val expr = constructor.newInstance(
+      child,
+      schemaValue,
+      optionsValue,
+      Boolean.box(charVarcharStandardSemantics),
+      Boolean.box(charVarcharFirstClassTypes))
     expr.asInstanceOf[Expression]
   }
 
   override def prettyName: String =
     getTagValue(FunctionRegistry.FUNC_ALIAS).getOrElse("from_avro")
+
+  // Persist flags are not user arguments; omit them from EXPLAIN / Connect goldens.
+  override protected def stringArgs: Iterator[Any] = super.stringArgs.take(3)
 }
 
 /**
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
index 4ab621e672df..8c5450882673 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
@@ -208,6 +208,30 @@ object CharVarcharUtils extends Logging with SparkCharVarcharUtils {
     }
   }
 
+  /**
+   * True if a CHAR/VARCHAR map key is not a valid identity key: CHAR(n) must be exactly n
+   * characters (shorter keys fail too); VARCHAR(n) must be at most n. STRING and other
+   * key types never violate. Avro map-key checks use this; JSON/XML keep local checks.
+   */
+  def charVarcharMapKeyViolatesLength(key: UTF8String, keyType: DataType): Boolean = {
+    keyType match {
+      case c: CharType => key.numChars() != c.length
+      case v: VarcharType => key.numChars() > v.length
+      case _ => false
+    }
+  }
+
+  /**
+   * Caps a map key echoed in an error so a long user-supplied name cannot blow the message.
+   */
+  def truncateMapKeyForError(key: UTF8String, maxChars: Int = 128): UTF8String = {
+    if (key != null && key.numChars() > maxChars) {
+      UTF8String.concat(key.substring(0, maxChars), UTF8String.fromString("..."))
+    } else {
+      key
+    }
+  }
+
   /**
    * Returns an expression to apply write-side string length check for the given expression. A
    * string value can not exceed N characters if it's written into a CHAR(N)/VARCHAR(N)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
index 1600a73e5519..95bfbe560d5d 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
@@ -45,7 +45,9 @@ import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
 import org.apache.spark.sql.catalyst.plans.logical.statsEstimation.ValueInterval
 import org.apache.spark.sql.catalyst.plans.physical.KeyReducer
 import org.apache.spark.sql.catalyst.trees.{Origin, TreeNode}
-import org.apache.spark.sql.catalyst.util.{sideBySide, CharsetProvider, DateTimeUtils, FailFastMode, IntervalUtils, MapData}
+import org.apache.spark.sql.catalyst.util.{
+  sideBySide, CharsetProvider, CharVarcharUtils, DateTimeUtils, FailFastMode, IntervalUtils, MapData
+}
 import org.apache.spark.sql.connector.catalog.{CatalogNotFoundException, Table, TableProvider}
 import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
 import org.apache.spark.sql.connector.expressions.Transform
@@ -2760,6 +2762,22 @@ private[sql] object QueryExecutionErrors extends QueryErrorsBase with ExecutionE
         "dataType" -> toSQLType(dataType)))
   }
 
+  /**
+   * Restricting CHAR/VARCHAR Avro map keys is a serde limitation (SQLSTATE 0A000),
+   * but this is a SparkRuntimeException so from_avro parse modes can handle it:
+   * PERMISSIVE nulls the record and FAILFAST surfaces the failure. Avro file scans
+   * do not apply per-row PERMISSIVE around this error.
+   */
+  def unsupportedAvroCharVarcharMapKey(
+      key: UTF8String, dataType: DataType): SparkRuntimeException = {
+    val displayKey = CharVarcharUtils.truncateMapKeyForError(key)
+    new SparkRuntimeException(
+      errorClass = "UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY",
+      messageParameters = Map(
+        "key" -> toSQLValue(displayKey, StringType),
+        "dataType" -> toSQLType(dataType)))
+  }
+
   def timestampAddOverflowError(micros: Long, amount: Long, unit: String): ArithmeticException = {
     new SparkArithmeticException(
       errorClass = "DATETIME_OVERFLOW",
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala b/sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
index b450d42ee7d8..3fde4aaad80e 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
@@ -34,10 +34,10 @@ import org.apache.spark.SparkRuntimeException
 import org.apache.spark.sql.avro.AvroUtils.{nonNullUnionBranches, toFieldStr, AvroMatchedField}
 import org.apache.spark.sql.catalyst.{InternalRow, NoopFilters, StructFilters}
 import org.apache.spark.sql.catalyst.expressions.{SpecificInternalRow, UnsafeArrayData}
-import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, ArrayData, DateTimeUtils, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, ArrayData, CharVarcharUtils, DateTimeUtils, GenericArrayData}
 import org.apache.spark.sql.catalyst.util.DateTimeConstants.MILLIS_PER_DAY
 import org.apache.spark.sql.catalyst.util.RebaseDateTime.RebaseSpec
-import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.errors.{QueryCompilationErrors, QueryExecutionErrors}
 import org.apache.spark.sql.execution.datasources.DataSourceUtils
 import org.apache.spark.sql.internal.{LegacyBehaviorPolicy, SQLConf}
 import org.apache.spark.sql.types._
@@ -46,12 +46,20 @@ import org.apache.spark.unsafe.types.UTF8String
 /**
  * A deserializer to deserialize data in avro format to data in catalyst format.
  *
+ * from_avro assigns CHAR/VARCHAR STRING values (pad, then trim-and-EXCEED_LIMIT_LENGTH).
+ * File scans leave that to ApplyCharTypePadding. ENUM symbols and to_avro encoding are
+ * unchanged. CHAR/VARCHAR map keys are length-checked without rewrite whenever
+ * standardSemantics is on (from_avro and scans), matching JSON/XML object names.
+ *
  * @param dataSchema The schema `rootCatalystType` was projected from, for a read that prunes
  *                   columns. A positional field match pairs a Catalyst field with the Avro field
  *                   at the same position, and that position is the one in the full schema rather
  *                   than in the projection: without this, reading only the third column would take
  *                   the first Avro field. `None` when the Catalyst type is not a projection;
  *                   unused when field matching is by name.
+ * @param applyCharVarcharAssignment When true (from_avro), assign CHAR/VARCHAR STRING values.
+ * @param charVarcharStandardSemanticsOverride Analysis-time standardSemantics for from_avro;
+ *                                             `None` reads the live session conf (file scans).
  */
 private[sql] class AvroDeserializer(
     rootAvroType: Schema,
@@ -62,7 +70,9 @@ private[sql] class AvroDeserializer(
     useStableIdForUnionType: Boolean,
     stableIdPrefixForUnionType: String,
     recursiveFieldMaxDepth: Int,
-    dataSchema: Option[StructType]) {
+    dataSchema: Option[StructType],
+    applyCharVarcharAssignment: Boolean = false,
+    charVarcharStandardSemanticsOverride: Option[Boolean] = None) {
 
   def this(
       rootAvroType: Schema,
@@ -83,6 +93,29 @@ private[sql] class AvroDeserializer(
       dataSchema = None)
   }
 
+  def this(
+      rootAvroType: Schema,
+      rootCatalystType: DataType,
+      datetimeRebaseMode: String,
+      useStableIdForUnionType: Boolean,
+      stableIdPrefixForUnionType: String,
+      recursiveFieldMaxDepth: Int,
+      applyCharVarcharAssignment: Boolean,
+      charVarcharStandardSemanticsOverride: Option[Boolean]) = {
+    this(
+      rootAvroType,
+      rootCatalystType,
+      positionalFieldMatch = false,
+      RebaseSpec(LegacyBehaviorPolicy.withName(datetimeRebaseMode)),
+      new NoopFilters,
+      useStableIdForUnionType,
+      stableIdPrefixForUnionType,
+      recursiveFieldMaxDepth,
+      dataSchema = None,
+      applyCharVarcharAssignment,
+      charVarcharStandardSemanticsOverride)
+  }
+
   private lazy val decimalConversions = new DecimalConversion()
 
   private val dateRebaseFunc = DataSourceUtils.createDateRebaseFuncInRead(
@@ -91,6 +124,10 @@ private[sql] class AvroDeserializer(
   private val timestampRebaseFunc = DataSourceUtils.createTimestampRebaseFuncInRead(
     datetimeRebaseSpec, "Avro")
 
+  // Lazy so converter setup cannot observe a default false if this field moves (JacksonParser).
+  private lazy val charVarcharStandardSemantics =
+    charVarcharStandardSemanticsOverride.getOrElse(SQLConf.get.charVarcharStandardSemantics)
+
   private val converter: Any => Option[Any] = try {
     rootCatalystType match {
       // A shortcut for empty schema.
@@ -260,7 +297,7 @@ private[sql] class AvroDeserializer(
       case (DOUBLE, DoubleType) => (updater, ordinal, value) =>
         updater.setDouble(ordinal, value.asInstanceOf[Double])
 
-      case (STRING, _: StringType) => (updater, ordinal, value) =>
+      case (STRING, st: StringType) => (updater, ordinal, value) =>
         val str = value match {
           case s: String => UTF8String.fromString(s)
           case s: Utf8 =>
@@ -268,7 +305,14 @@ private[sql] class AvroDeserializer(
             System.arraycopy(s.getBytes, 0, bytes, 0, s.getByteLength)
             UTF8String.fromBytes(bytes)
         }
-        updater.set(ordinal, str)
+        // File scans leave CHAR padding to ApplyCharTypePadding. from_avro assigns STRING
+        // values only; ENUM symbols stay as-is so a padded CHAR cannot invent a symbol.
+        val assigned = if (applyCharVarcharAssignment) {
+          CharVarcharUtils.applyTextParseSemantics(str, st)
+        } else {
+          str
+        }
+        updater.set(ordinal, assigned)
 
       case (ENUM, _: StringType) => (updater, ordinal, value) =>
         updater.set(ordinal, UTF8String.fromString(value.toString))
@@ -356,10 +400,13 @@ private[sql] class AvroDeserializer(
           updater.set(ordinal, result)
 
       case (MAP, MapType(keyType: StringType, valueType, valueContainsNull)) =>
-        val keyWriter = newWriter(SchemaBuilder.builder().stringType(), keyType,
+        // Decode keys as unbounded STRING so CHAR/VARCHAR assignment never rewrites them.
+        // Identity length-check matches JSON/XML map keys (no pad/trim).
+        val keyWriter = newWriter(SchemaBuilder.builder().stringType(), StringType,
           avroPath :+ "key", catalystPath :+ "key")
         val valueWriter = newWriter(avroType.getValueType, valueType,
           avroPath :+ "value", catalystPath :+ "value")
+        val mapKeyChecker = makeAvroMapKeyChecker(keyType)
         (updater, ordinal, value) =>
           val map = value.asInstanceOf[java.util.Map[AnyRef, AnyRef]]
           val keyArray = createArrayData(keyType, map.size())
@@ -372,6 +419,7 @@ private[sql] class AvroDeserializer(
             val entry = iter.next()
             assert(entry.getKey != null)
             keyWriter(keyUpdater, i, entry.getKey)
+            mapKeyChecker.foreach(_(keyArray.getUTF8String(i)))
             if (entry.getValue == null) {
               if (!valueContainsNull) {
                 throw new SparkRuntimeException(
@@ -530,6 +578,28 @@ private[sql] class AvroDeserializer(
     }
   }
 
+  /**
+   * Length-check CHAR/VARCHAR Avro map keys without pad/trim. Built once per map converter.
+   * Gated on standardSemantics only (not from_avro assignment): scans must reject off-width
+   * keys too, otherwise ApplyCharTypePadding can rewrite CHAR keys and HashMap.put last-wins.
+   * UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY lets from_avro PERMISSIVE swallow the whole record.
+   */
+  private def makeAvroMapKeyChecker(keyType: DataType): Option[UTF8String => Unit] = {
+    if (!charVarcharStandardSemantics) {
+      None
+    } else {
+      keyType match {
+        case _: CharType | _: VarcharType =>
+          Some { key =>
+            if (CharVarcharUtils.charVarcharMapKeyViolatesLength(key, keyType)) {
+              throw QueryExecutionErrors.unsupportedAvroCharVarcharMapKey(key, keyType)
+            }
+          }
+        case _ => None
+      }
+    }
+  }
+
   private def createArrayData(elementType: DataType, length: Int): ArrayData = elementType match {
     case BooleanType => UnsafeArrayData.fromPrimitiveArray(new Array[Boolean](length))
     case ByteType => UnsafeArrayData.fromPrimitiveArray(new Array[Byte](length))
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
index fa091a8d4b32..19f94d2f6b84 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
@@ -2234,6 +2234,7 @@ class BasicCharVarcharTestSuite extends SharedSparkSession {
       withTempPath { file =>
         val ser = new org.apache.spark.sql.avro.AvroSerializer(
           nestedSchema, nestedAvro, nullable = false)
+        // File-scan deserializers do not assign CHAR padding; values are stored padded.
         val row = org.apache.spark.sql.catalyst.InternalRow(
           org.apache.spark.unsafe.types.UTF8String.fromString("ab  "),
           org.apache.spark.sql.catalyst.InternalRow(
@@ -2272,6 +2273,73 @@ class BasicCharVarcharTestSuite extends SharedSparkSession {
         }
       }
 
+      // SPARK-60242: from_avro-equivalent decode assigns CHAR/VARCHAR STRING values.
+      // File-scan deserializers (default) and to_avro encoding do not.
+      Seq(CharType(4), VarcharType(4)).foreach { dt =>
+        val avro = converters.toAvroType(dt, nullable = false)
+        val fromAvroDeser = new org.apache.spark.sql.avro.AvroDeserializer(
+          avro, dt, "CORRECTED", false, "", -1,
+          applyCharVarcharAssignment = true,
+          charVarcharStandardSemanticsOverride = None)
+        val scanDeser = new org.apache.spark.sql.avro.AvroDeserializer(
+          avro, dt, "CORRECTED", false, "", -1)
+        val padded = dt match {
+          case _: CharType => "ab  "
+          case _ => "ab"
+        }
+        assert(
+          fromAvroDeser.deserialize(new org.apache.avro.util.Utf8("ab")).get.toString === padded,
+          s"from_avro-equivalent decode should assign $dt")
+        assert(
+          scanDeser.deserialize(new org.apache.avro.util.Utf8("ab")).get.toString === "ab",
+          s"file-scan decode should not assign $dt")
+        checkError(
+          exception = intercept[SparkRuntimeException] {
+            fromAvroDeser.deserialize(new org.apache.avro.util.Utf8("abcde"))
+          },
+          condition = "EXCEED_LIMIT_LENGTH",
+          parameters = Map("limit" -> "4"))
+        val ser = new org.apache.spark.sql.avro.AvroSerializer(dt, avro, nullable = false)
+        val encoded = ser.serialize(
+          org.apache.spark.unsafe.types.UTF8String.fromString("ab"))
+          .asInstanceOf[org.apache.avro.util.Utf8]
+        assert(encoded.toString === "ab", s"to_avro-equivalent encode should not assign $dt")
+      }
+
+      val enumSchema = org.apache.avro.Schema.createEnum(
+        "Spark60242Enum", "", "", java.util.Arrays.asList("ab", "cd"))
+      val enumVal = new org.apache.avro.generic.GenericData.EnumSymbol(enumSchema, "ab")
+      val enumDeser = new org.apache.spark.sql.avro.AvroDeserializer(
+        enumSchema, CharType(4), "CORRECTED", false, "", -1,
+        applyCharVarcharAssignment = true,
+        charVarcharStandardSemanticsOverride = None)
+      assert(
+        enumDeser.deserialize(enumVal).get.toString === "ab",
+        "ENUM symbols must not be CHAR-padded")
+
+      val mapDt = MapType(CharType(2), IntegerType)
+      val mapAvro = converters.toAvroType(mapDt, nullable = false)
+      val shortKeyMap = new java.util.HashMap[AnyRef, AnyRef]()
+      shortKeyMap.put(new org.apache.avro.util.Utf8("a"), Integer.valueOf(1))
+      withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "false") {
+        val captured = new org.apache.spark.sql.avro.AvroDeserializer(
+          mapAvro, mapDt, "CORRECTED", false, "", -1,
+          applyCharVarcharAssignment = false,
+          charVarcharStandardSemanticsOverride = Some(true))
+        checkError(
+          exception = intercept[SparkRuntimeException] {
+            captured.deserialize(shortKeyMap)
+          },
+          condition = "UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY",
+          sqlState = "0A000",
+          parameters = Map("key" -> "'a'", "dataType" -> "\"CHAR(2)\""))
+        val live = new org.apache.spark.sql.avro.AvroDeserializer(
+          mapAvro, mapDt, "CORRECTED", false, "", -1)
+        val liveMap = live.deserialize(shortKeyMap).get
+          .asInstanceOf[org.apache.spark.sql.catalyst.util.MapData]
+        assert(liveMap.keyArray().getUTF8String(0).toString === "a")
+      }
+
       // JSON has no embedded schema, so a user-specified schema supplies the logical type.
       withTempPath { dir =>
         val path = dir.getCanonicalPath


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org