You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by dt...@apache.org on 2026/10/16 15:54:10 UTC
(spark) branch branch-4.x updated: [SPARK-60242][SQL] Assign CHAR/VARCHAR in from_avro and identity-check Avro map keys
This is an automated email from the ASF dual-hosted git repository.
dtenedor pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new ff82b485cce7 [SPARK-60242][SQL] Assign CHAR/VARCHAR in from_avro and identity-check Avro map keys
ff82b485cce7 is described below
commit ff82b485cce7ca6746fd9759bcd800c729d972a9
Author: Serge Rielau <se...@rielau.com>
AuthorDate: Fri Oct 16 08:53:49 2026 -0700
[SPARK-60242][SQL] Assign CHAR/VARCHAR in from_avro and identity-check Avro map keys
### What changes were proposed in this pull request?
CHAR/VARCHAR for Avro under SPARK-58794 (SPARK-60242). Behavior matches the JSON/XML matrix; this PR does not change the JSON or XML parsers.
Contract:
- `from_avro` assigns CHAR/VARCHAR on STRING values (pad/trim, `EXCEED_LIMIT_LENGTH`). Avro ENUM symbols are not assigned.
- `to_avro` does not assign CHAR/VARCHAR.
- Avro file scans do not assign STRING values (ApplyCharTypePadding still runs for CHAR columns). MAP CHAR/VARCHAR keys are identity-checked so CHAR padding cannot last-win-merge distinct keys.
- MAP CHAR/VARCHAR keys in `from_avro` and scans use an identity length-check (`UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY`). Keys are not padded or trimmed.
- PERMISSIVE nulls a malformed record only in `from_avro` (`AvroDataToCatalyst`). File scans do not wrap per-row PERMISSIVE around that error.
- `FromAvro` and `AvroDataToCatalyst` persist `charVarcharStandardSemantics` and `charVarcharFirstClassTypes` at analysis so a view keeps creation-time types and map-key checks after SET.
### Why are the changes needed?
Avro already stamped CHAR/VARCHAR from `spark.sql.catalyst.type` but copied UTF-8 bytes. Without assignment, `from_avro` CHAR values were unpadded; without identity MAP-key checks, file-scan CHAR keys could be padded and collide.
### Does this PR introduce _any_ user-facing change?
Yes, when `spark.sql.charVarchar.standardSemantics.enabled` is true (first-class CHAR/VARCHAR):
- `from_avro` into CHAR/VARCHAR pads/trims STRING values and rejects oversize values with `EXCEED_LIMIT_LENGTH` (the whole record is null in PERMISSIVE).
- MAP keys of CHAR/VARCHAR that are not exactly the declared length fail with `UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY`.
- Avro file scans with MAP CHAR/VARCHAR keys reject off-width keys rather than padding them into collisions.
- `to_avro` does not pad or trim.
This is user-facing on unreleased branches; it is not a change to a released Spark version.
### How was this patch tested?
- `avro/testOnly *AvroFunctionsSuite -- -z SPARK-60242`
- `avro/testOnly org.apache.spark.sql.avro.AvroV1Suite org.apache.spark.sql.avro.AvroV2Suite -- -z SPARK-58814`
- `JAVA_HOME=... COURSIER_REPOSITORIES=... ./dev/lint-scala`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Grok 4.6
Closes #59468 from srielau/SPARK-60242.
Authored-by: Serge Rielau <se...@rielau.com>
Signed-off-by: Daniel Tenedorio <da...@databricks.com>
(cherry picked from commit 6c3b628d8a978e0c0d0e54a9b7767fc62709a966)
Signed-off-by: Daniel Tenedorio <da...@databricks.com>
---
.../src/main/resources/error/error-conditions.json | 7 +
.../apache/spark/sql/avro/AvroDataToCatalyst.scala | 44 +++-
.../apache/spark/sql/avro/AvroFunctionsSuite.scala | 249 ++++++++++++++++++++-
.../catalyst/expressions/avroSqlFunctions.scala | 37 ++-
.../spark/sql/catalyst/util/CharVarcharUtils.scala | 24 ++
.../spark/sql/errors/QueryExecutionErrors.scala | 20 +-
.../apache/spark/sql/avro/AvroDeserializer.scala | 82 ++++++-
.../apache/spark/sql/CharVarcharTestSuite.scala | 68 ++++++
8 files changed, 514 insertions(+), 17 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json
index 8a2dd3bd44ac..f7e265f5969e 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -8783,6 +8783,13 @@
},
"sqlState" : "42000"
},
+ "UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY" : {
+ "message" : [
+ "The Avro map key <key> is not a valid key for the map key type <dataType>.",
+ "A CHAR key must match the declared length exactly and a VARCHAR key must not exceed it; keys are never padded or trimmed. Use a STRING map key to accept any name."
+ ],
+ "sqlState" : "0A000"
+ },
"UNSUPPORTED_BUILT_IN_JDBC_DIALECT" : {
"message" : [
"`<name>` is not a supported built-in JDBC dialect. Supported dialects are: <supportedNames>."
diff --git a/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala b/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala
index 87bb2f2a17c2..17e82804d6e3 100644
--- a/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala
+++ b/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala
@@ -25,24 +25,47 @@ import org.apache.avro.io.{BinaryDecoder, DecoderFactory}
import org.apache.spark.SparkException
import org.apache.spark.sql.catalyst.expressions.{ExpectsInputTypes, Expression, SpecificInternalRow, UnaryExpression}
import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, CodeGenerator, ExprCode}
-import org.apache.spark.sql.catalyst.util.{FailFastMode, ParseMode, PermissiveMode}
+import org.apache.spark.sql.catalyst.util.{CharVarcharUtils, FailFastMode, ParseMode, PermissiveMode}
import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
case class AvroDataToCatalyst(
child: Expression,
jsonFormatSchema: String,
- options: Map[String, String])
+ options: Map[String, String],
+ // PERSISTED: capture at analysis (like JsonToStructs) so copy()/views keep
+ // creation-time CHAR/VARCHAR map-key checks, not the caller's session setting.
+ charVarcharStandardSemantics: Boolean,
+ charVarcharFirstClassTypes: Boolean)
extends UnaryExpression with ExpectsInputTypes {
+ // Tests and 3-arg apply capture the current session flags.
+ def this(child: Expression, jsonFormatSchema: String, options: Map[String, String]) =
+ this(
+ child,
+ jsonFormatSchema,
+ options,
+ SQLConf.get.charVarcharStandardSemantics,
+ SQLConf.get.charVarcharFirstClassTypes)
+
override def inputTypes: Seq[AbstractDataType] = Seq(BinaryType)
override lazy val dataType: DataType = {
- val dt = SchemaConverters.toSqlType(
+ val converted = SchemaConverters.toSqlType(
expectedSchema,
avroOptions.useStableIdForUnionType,
avroOptions.stableIdPrefixForUnionType,
avroOptions.recursiveFieldMaxDepth).dataType
+ // SchemaConverters restores CHAR/VARCHAR stamps from Avro even when first-class types
+ // are off. Use the analysis-time snapshot so copy() cannot drop CHAR map keys.
+ val dt = if (charVarcharFirstClassTypes) {
+ converted
+ } else {
+ // Conf-independent: replaceCharVarcharWithString would keep CHAR/VARCHAR if the
+ // live session has first-class types on.
+ CharVarcharUtils.replaceCharVarcharWithStringAlways(converted)
+ }
parseMode match {
// With PermissiveMode, the output Catalyst row might contain columns of null values for
// corrupt records, even if some of the columns are not nullable in the user-provided schema.
@@ -69,7 +92,9 @@ case class AvroDataToCatalyst(
avroOptions.datetimeRebaseModeInRead,
avroOptions.useStableIdForUnionType,
avroOptions.stableIdPrefixForUnionType,
- avroOptions.recursiveFieldMaxDepth)
+ avroOptions.recursiveFieldMaxDepth,
+ applyCharVarcharAssignment = true,
+ charVarcharStandardSemanticsOverride = Some(charVarcharStandardSemantics))
@transient private var decoder: BinaryDecoder = _
@@ -127,6 +152,9 @@ case class AvroDataToCatalyst(
override def prettyName: String = "from_avro"
+ // Persist flags are not user arguments; omit them from EXPLAIN / Connect goldens.
+ override protected def stringArgs: Iterator[Any] = super.stringArgs.take(3)
+
override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
val expr = ctx.addReferenceObj("this", this)
nullSafeCodeGen(ctx, ev, eval => {
@@ -146,3 +174,11 @@ case class AvroDataToCatalyst(
override protected def withNewChildInternal(newChild: Expression): AvroDataToCatalyst =
copy(child = newChild)
}
+
+object AvroDataToCatalyst {
+ def apply(
+ child: Expression,
+ jsonFormatSchema: String,
+ options: Map[String, String]): AvroDataToCatalyst =
+ new AvroDataToCatalyst(child, jsonFormatSchema, options)
+}
diff --git a/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala b/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala
index bcc104aa3e82..b30e92c28731 100644
--- a/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala
+++ b/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroFunctionsSuite.scala
@@ -26,15 +26,17 @@ import org.apache.avro.file.SeekableByteArrayInput
import org.apache.avro.generic.{GenericDatumReader, GenericDatumWriter, GenericRecord, GenericRecordBuilder}
import org.apache.avro.io.{DecoderFactory, EncoderFactory}
-import org.apache.spark.SparkException
+import org.apache.spark.{SparkException, SparkRuntimeException}
import org.apache.spark.sql.{AnalysisException, Row}
import org.apache.spark.sql.avro.{functions => Fns}
import org.apache.spark.sql.avro.functions.{from_avro, to_avro}
import org.apache.spark.sql.execution.LocalTableScanExec
-import org.apache.spark.sql.functions.{col, lit, struct}
+import org.apache.spark.sql.functions.{col, lit, map, struct}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
-import org.apache.spark.sql.types.{BinaryType, IntegerType, StructField, StructType}
+import org.apache.spark.sql.types.{
+ BinaryType, CharType, IntegerType, MapType, StringType, StructField, StructType, VarcharType
+}
class AvroFunctionsSuite extends SharedSparkSession {
import testImplicits._
@@ -769,6 +771,247 @@ class AvroFunctionsSuite extends SharedSparkSession {
s"Should not throw NPE, but got: ${ex.getClass.getName}: ${ex.getMessage}")
}
+ test("SPARK-60242: from_avro CHAR/VARCHAR length enforcement") {
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ val stringSchema = """{"type": "string"}"""
+ val charSchema = SchemaConverters.toAvroType(CharType(4), nullable = false).toString
+ val varcharSchema = SchemaConverters.toAvroType(VarcharType(4), nullable = false).toString
+ val stringMapSchema =
+ SchemaConverters.toAvroType(MapType(StringType, IntegerType), nullable = false).toString
+ val charMapSchema =
+ SchemaConverters.toAvroType(MapType(CharType(2), IntegerType), nullable = false).toString
+ val varcharMapSchema =
+ SchemaConverters.toAvroType(
+ MapType(VarcharType(3), IntegerType), nullable = false).toString
+
+ val padded = spark.range(1).select(
+ from_avro(to_avro(lit("ab"), stringSchema), charSchema).as("c"))
+ assert(padded.schema.head.dataType === CharType(4))
+ checkAnswer(padded.selectExpr("concat('<', c, '>')"), Row("<ab >"))
+
+ val varcharShort = spark.range(1).select(
+ from_avro(to_avro(lit("ab"), stringSchema), varcharSchema).as("v"))
+ assert(varcharShort.schema.head.dataType === VarcharType(4))
+ checkAnswer(varcharShort, Row("ab"))
+
+ val roundTrip = spark.range(1).selectExpr("cast('ab' AS CHAR(4)) AS c")
+ .select(from_avro(to_avro($"c", charSchema), charSchema).as("c"))
+ checkAnswer(roundTrip.selectExpr("concat('<', c, '>')"), Row("<ab >"))
+
+ def assertExceedLimit(body: => Any): Unit = {
+ val e = intercept[SparkException] { body }
+ val cause = e.getCause match {
+ case r: SparkRuntimeException => r
+ case other =>
+ Option(other).flatMap(t => Option(t.getCause)).getOrElse(other) match {
+ case r: SparkRuntimeException => r
+ case _ => fail(s"expected EXCEED_LIMIT_LENGTH cause, got: $e")
+ }
+ }
+ checkError(
+ exception = cause,
+ condition = "EXCEED_LIMIT_LENGTH",
+ parameters = Map("limit" -> "4"))
+ }
+ val oversize = spark.range(1).select(to_avro(lit("abcde"), stringSchema).as("bin"))
+ assertExceedLimit(oversize.select(from_avro($"bin", charSchema)).collect())
+ assertExceedLimit(oversize.select(from_avro($"bin", varcharSchema)).collect())
+
+ val permissive = oversize.select(
+ from_avro($"bin", charSchema, Map("mode" -> "PERMISSIVE").asJava))
+ checkAnswer(permissive, Row(null))
+
+ def assertUnsupportedAvroMapKey(
+ body: => Any, key: String, dataTypeSql: String): Unit = {
+ val error = intercept[Exception] { body }
+ val cause = Iterator.iterate[Throwable](error)(_.getCause)
+ .takeWhile(_ != null)
+ .collectFirst {
+ case e: SparkRuntimeException
+ if e.getCondition == "UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY" => e
+ }
+ .getOrElse(
+ fail("expected UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY cause", error))
+ checkError(
+ exception = cause,
+ condition = "UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY",
+ sqlState = "0A000",
+ parameters = Map("key" -> s"'$key'", "dataType" -> s""""$dataTypeSql""""))
+ }
+
+ val exactCharKey = spark.range(1).select(
+ from_avro(
+ to_avro(map(lit("ab"), lit(1)), stringMapSchema), charMapSchema).as("m"))
+ assert(exactCharKey.schema.head.dataType.asInstanceOf[MapType].keyType ===
+ CharType(2))
+ checkAnswer(exactCharKey, Row(Map("ab" -> 1)))
+
+ assertUnsupportedAvroMapKey(
+ spark.range(1).select(
+ from_avro(
+ to_avro(map(lit("a"), lit(1)), stringMapSchema), charMapSchema)).collect(),
+ "a",
+ "CHAR(2)")
+ assertUnsupportedAvroMapKey(
+ spark.range(1).select(
+ from_avro(
+ to_avro(map(lit("abc"), lit(1)), stringMapSchema), charMapSchema)).collect(),
+ "abc",
+ "CHAR(2)")
+
+ val varcharKey = spark.range(1).select(
+ from_avro(
+ to_avro(map(lit("ab"), lit(1)), stringMapSchema), varcharMapSchema).as("m"))
+ checkAnswer(varcharKey, Row(Map("ab" -> 1)))
+ val varcharSpacedKey = spark.range(1).select(
+ from_avro(
+ to_avro(map(lit("ab "), lit(1)), stringMapSchema), varcharMapSchema).as("m"))
+ checkAnswer(varcharSpacedKey, Row(Map("ab " -> 1)))
+ assertUnsupportedAvroMapKey(
+ spark.range(1).select(
+ from_avro(
+ to_avro(map(lit("abcd"), lit(1)), stringMapSchema),
+ varcharMapSchema)).collect(),
+ "abcd",
+ "VARCHAR(3)")
+
+ checkAnswer(
+ spark.range(1).select(
+ from_avro(
+ to_avro(map(lit("a"), lit(1)), stringMapSchema),
+ charMapSchema,
+ Map("mode" -> "PERMISSIVE").asJava)),
+ Row(null))
+
+ val char0MapSchema =
+ SchemaConverters.toAvroType(MapType(CharType(0), IntegerType), nullable = false)
+ .toString
+ checkAnswer(
+ spark.range(1).select(
+ from_avro(
+ to_avro(map(lit(""), lit(1)), stringMapSchema), char0MapSchema).as("m")),
+ Row(Map("" -> 1)))
+ assertUnsupportedAvroMapKey(
+ spark.range(1).select(
+ from_avro(
+ to_avro(map(lit("a"), lit(1)), stringMapSchema), char0MapSchema)).collect(),
+ "a",
+ "CHAR(0)")
+
+ val stringStructSchema =
+ SchemaConverters.toAvroType(
+ new StructType().add("c", StringType), nullable = false).toString
+ val charStructSchema =
+ SchemaConverters.toAvroType(
+ new StructType().add("c", CharType(4)), nullable = false).toString
+ checkAnswer(
+ spark.range(1).select(
+ from_avro(
+ to_avro(struct(lit("ab").as("c")), stringStructSchema),
+ charStructSchema).as("s"))
+ .selectExpr("concat('<', s.c, '>')"),
+ Row("<ab >"))
+
+ withTempView("avro_cv_map_view") {
+ spark.range(1).select(
+ from_avro(
+ to_avro(map(lit("a"), lit(1)), stringMapSchema), charMapSchema).as("m"))
+ .createOrReplaceTempView("avro_cv_map_view")
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "false") {
+ assertUnsupportedAvroMapKey(
+ spark.sql("SELECT * FROM avro_cv_map_view").collect(),
+ "a",
+ "CHAR(2)")
+ }
+ }
+
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+ spark.range(1).select(
+ map(lit("a"), lit(1), lit("a "), lit(2)).as("m"))
+ .write.mode("overwrite").format("avro").save(path)
+ assertUnsupportedAvroMapKey(
+ spark.read.schema("m MAP<CHAR(2), INT>").format("avro").load(path).collect(),
+ "a",
+ "CHAR(2)")
+ }
+
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+ spark.range(1).select(
+ map(lit("ab"), lit(1), lit("a "), lit(2)).as("m"))
+ .write.mode("overwrite").format("avro").save(path)
+ checkAnswer(
+ spark.read.schema("m MAP<CHAR(2), INT>").format("avro").load(path),
+ Row(Map("ab" -> 1, "a " -> 2)))
+ }
+
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+ spark.range(1).select(map(lit("ab"), lit("xy")).as("m"))
+ .write.mode("overwrite").format("avro").save(path)
+ val nested = spark.read.schema("m MAP<CHAR(2), CHAR(4)>").format("avro").load(path)
+ checkAnswer(nested.selectExpr("concat('<', m['ab'], '>')"), Row("<xy >"))
+ }
+
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+ spark.range(1).select(map(lit("abcd"), lit(1)).as("m"))
+ .write.mode("overwrite").format("avro").save(path)
+ assertUnsupportedAvroMapKey(
+ spark.read.schema("m MAP<VARCHAR(3), INT>").format("avro").load(path).collect(),
+ "abcd",
+ "VARCHAR(3)")
+ }
+ }
+
+ withSQLConf(
+ SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "false",
+ SQLConf.PRESERVE_CHAR_VARCHAR_TYPE_INFO.key -> "true") {
+ val stringSchema = """{"type": "string"}"""
+ val charSchema = SchemaConverters.toAvroType(CharType(4), nullable = false).toString
+ val stringMapSchema =
+ SchemaConverters.toAvroType(MapType(StringType, IntegerType), nullable = false).toString
+ val charMapSchema =
+ SchemaConverters.toAvroType(MapType(CharType(2), IntegerType), nullable = false).toString
+ val padded = spark.range(1).select(
+ from_avro(to_avro(lit("ab"), stringSchema), charSchema).as("c"))
+ assert(padded.schema.head.dataType === CharType(4))
+ checkAnswer(padded.selectExpr("concat('<', c, '>')"), Row("<ab >"))
+ // Off-width CHAR keys are not rejected or rewritten when standardSemantics is off.
+ // Exclude ConstantFolding so a CHAR Cast cannot pad the folded map key. Collecting
+ // MAP<CHAR(n), _> would still pad via the encoder, so to_json shows the raw key.
+ withSQLConf(SQLConf.OPTIMIZER_EXCLUDED_RULES.key ->
+ "org.apache.spark.sql.catalyst.optimizer.ConstantFolding") {
+ val shortKey = spark.range(1).select(
+ from_avro(
+ to_avro(map(lit("a"), lit(1)), stringMapSchema), charMapSchema).as("m"))
+ assert(shortKey.schema.head.dataType.asInstanceOf[MapType].keyType === CharType(2))
+ checkAnswer(shortKey.selectExpr("to_json(m)"), Row("""{"a":1}"""))
+ }
+ }
+
+ withSQLConf(
+ SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "false",
+ SQLConf.PRESERVE_CHAR_VARCHAR_TYPE_INFO.key -> "false") {
+ val stringSchema = """{"type": "string"}"""
+ val charSchema = SchemaConverters.toAvroType(CharType(4), nullable = false).toString
+ val df = spark.range(1).select(
+ from_avro(to_avro(lit("abcdef"), stringSchema), charSchema).as("c"))
+ assert(df.schema.head.dataType === StringType)
+ checkAnswer(df, Row("abcdef"))
+
+ withTempView("avro_cv_string_view") {
+ df.createOrReplaceTempView("avro_cv_string_view")
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ val fromView = spark.sql("SELECT * FROM avro_cv_string_view")
+ assert(fromView.schema.head.dataType === StringType)
+ checkAnswer(fromView, Row("abcdef"))
+ }
+ }
+ }
+ }
+
test("SPARK-56043: from_avro with completely unparseable schema should not throw NPE") {
val garbageSchema = "this is not valid JSON at all"
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala
index cb553aeb96ff..52b5ee67dd42 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala
@@ -21,6 +21,7 @@ import org.apache.spark.sql.catalyst.analysis.FunctionRegistry
import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.util.ArrayBasedMapData
import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{MapType, NullType, StringType}
import org.apache.spark.unsafe.types.UTF8String
import org.apache.spark.util.Utils
@@ -62,12 +63,29 @@ import org.apache.spark.util.Utils
since = "4.0.0"
)
// scalastyle:on line.size.limit
-case class FromAvro(child: Expression, jsonFormatSchema: Expression, options: Expression)
+case class FromAvro(
+ child: Expression,
+ jsonFormatSchema: Expression,
+ options: Expression,
+ // PERSISTED: copy()/views keep analysis-time CHAR/VARCHAR flags. FromAvro is
+ // RuntimeReplaceable, so a new replacement would otherwise re-read SQLConf.
+ // FunctionRegistry matches Expression-only ctors; keep 2-arg and 3-arg.
+ charVarcharStandardSemantics: Boolean,
+ charVarcharFirstClassTypes: Boolean)
extends TernaryExpression with RuntimeReplaceable {
override def first: Expression = child
override def second: Expression = jsonFormatSchema
override def third: Expression = options
+ // FunctionRegistry matches Expression-only ctors; keep 2-arg and 3-arg.
+ def this(child: Expression, jsonFormatSchema: Expression, options: Expression) =
+ this(
+ child,
+ jsonFormatSchema,
+ options,
+ SQLConf.get.charVarcharStandardSemantics,
+ SQLConf.get.charVarcharFirstClassTypes)
+
def this(child: Expression, jsonFormatSchema: Expression) =
this(child, jsonFormatSchema, Literal.create(null))
@@ -120,17 +138,30 @@ case class FromAvro(child: Expression, jsonFormatSchema: Expression, options: Ex
Map.empty
}
val constructor = try {
- Utils.classForName("org.apache.spark.sql.avro.AvroDataToCatalyst").getConstructors().head
+ // 5-arg primary: child, schema, options, plus the two persisted CHAR/VARCHAR flags.
+ Utils.classForName("org.apache.spark.sql.avro.AvroDataToCatalyst").getConstructors
+ .find { c =>
+ val t = c.getParameterTypes
+ t.length == 5 && t(3) == classOf[Boolean] && t(4) == classOf[Boolean]
+ }.get
} catch {
case _: java.lang.ClassNotFoundException =>
throw QueryCompilationErrors.avroNotLoadedSqlFunctionsUnusable(functionName = "FROM_AVRO")
}
- val expr = constructor.newInstance(child, schemaValue, optionsValue)
+ val expr = constructor.newInstance(
+ child,
+ schemaValue,
+ optionsValue,
+ Boolean.box(charVarcharStandardSemantics),
+ Boolean.box(charVarcharFirstClassTypes))
expr.asInstanceOf[Expression]
}
override def prettyName: String =
getTagValue(FunctionRegistry.FUNC_ALIAS).getOrElse("from_avro")
+
+ // Persist flags are not user arguments; omit them from EXPLAIN / Connect goldens.
+ override protected def stringArgs: Iterator[Any] = super.stringArgs.take(3)
}
/**
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
index 4ab621e672df..8c5450882673 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
@@ -208,6 +208,30 @@ object CharVarcharUtils extends Logging with SparkCharVarcharUtils {
}
}
+ /**
+ * True if a CHAR/VARCHAR map key is not a valid identity key: CHAR(n) must be exactly n
+ * characters (shorter keys fail too); VARCHAR(n) must be at most n. STRING and other
+ * key types never violate. Avro map-key checks use this; JSON/XML keep local checks.
+ */
+ def charVarcharMapKeyViolatesLength(key: UTF8String, keyType: DataType): Boolean = {
+ keyType match {
+ case c: CharType => key.numChars() != c.length
+ case v: VarcharType => key.numChars() > v.length
+ case _ => false
+ }
+ }
+
+ /**
+ * Caps a map key echoed in an error so a long user-supplied name cannot blow the message.
+ */
+ def truncateMapKeyForError(key: UTF8String, maxChars: Int = 128): UTF8String = {
+ if (key != null && key.numChars() > maxChars) {
+ UTF8String.concat(key.substring(0, maxChars), UTF8String.fromString("..."))
+ } else {
+ key
+ }
+ }
+
/**
* Returns an expression to apply write-side string length check for the given expression. A
* string value can not exceed N characters if it's written into a CHAR(N)/VARCHAR(N)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
index 1600a73e5519..95bfbe560d5d 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
@@ -45,7 +45,9 @@ import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.spark.sql.catalyst.plans.logical.statsEstimation.ValueInterval
import org.apache.spark.sql.catalyst.plans.physical.KeyReducer
import org.apache.spark.sql.catalyst.trees.{Origin, TreeNode}
-import org.apache.spark.sql.catalyst.util.{sideBySide, CharsetProvider, DateTimeUtils, FailFastMode, IntervalUtils, MapData}
+import org.apache.spark.sql.catalyst.util.{
+ sideBySide, CharsetProvider, CharVarcharUtils, DateTimeUtils, FailFastMode, IntervalUtils, MapData
+}
import org.apache.spark.sql.connector.catalog.{CatalogNotFoundException, Table, TableProvider}
import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
import org.apache.spark.sql.connector.expressions.Transform
@@ -2760,6 +2762,22 @@ private[sql] object QueryExecutionErrors extends QueryErrorsBase with ExecutionE
"dataType" -> toSQLType(dataType)))
}
+ /**
+ * Restricting CHAR/VARCHAR Avro map keys is a serde limitation (SQLSTATE 0A000),
+ * but this is a SparkRuntimeException so from_avro parse modes can handle it:
+ * PERMISSIVE nulls the record and FAILFAST surfaces the failure. Avro file scans
+ * do not apply per-row PERMISSIVE around this error.
+ */
+ def unsupportedAvroCharVarcharMapKey(
+ key: UTF8String, dataType: DataType): SparkRuntimeException = {
+ val displayKey = CharVarcharUtils.truncateMapKeyForError(key)
+ new SparkRuntimeException(
+ errorClass = "UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY",
+ messageParameters = Map(
+ "key" -> toSQLValue(displayKey, StringType),
+ "dataType" -> toSQLType(dataType)))
+ }
+
def timestampAddOverflowError(micros: Long, amount: Long, unit: String): ArithmeticException = {
new SparkArithmeticException(
errorClass = "DATETIME_OVERFLOW",
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala b/sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
index b450d42ee7d8..3fde4aaad80e 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
@@ -34,10 +34,10 @@ import org.apache.spark.SparkRuntimeException
import org.apache.spark.sql.avro.AvroUtils.{nonNullUnionBranches, toFieldStr, AvroMatchedField}
import org.apache.spark.sql.catalyst.{InternalRow, NoopFilters, StructFilters}
import org.apache.spark.sql.catalyst.expressions.{SpecificInternalRow, UnsafeArrayData}
-import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, ArrayData, DateTimeUtils, GenericArrayData}
+import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, ArrayData, CharVarcharUtils, DateTimeUtils, GenericArrayData}
import org.apache.spark.sql.catalyst.util.DateTimeConstants.MILLIS_PER_DAY
import org.apache.spark.sql.catalyst.util.RebaseDateTime.RebaseSpec
-import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.errors.{QueryCompilationErrors, QueryExecutionErrors}
import org.apache.spark.sql.execution.datasources.DataSourceUtils
import org.apache.spark.sql.internal.{LegacyBehaviorPolicy, SQLConf}
import org.apache.spark.sql.types._
@@ -46,12 +46,20 @@ import org.apache.spark.unsafe.types.UTF8String
/**
* A deserializer to deserialize data in avro format to data in catalyst format.
*
+ * from_avro assigns CHAR/VARCHAR STRING values (pad, then trim-and-EXCEED_LIMIT_LENGTH).
+ * File scans leave that to ApplyCharTypePadding. ENUM symbols and to_avro encoding are
+ * unchanged. CHAR/VARCHAR map keys are length-checked without rewrite whenever
+ * standardSemantics is on (from_avro and scans), matching JSON/XML object names.
+ *
* @param dataSchema The schema `rootCatalystType` was projected from, for a read that prunes
* columns. A positional field match pairs a Catalyst field with the Avro field
* at the same position, and that position is the one in the full schema rather
* than in the projection: without this, reading only the third column would take
* the first Avro field. `None` when the Catalyst type is not a projection;
* unused when field matching is by name.
+ * @param applyCharVarcharAssignment When true (from_avro), assign CHAR/VARCHAR STRING values.
+ * @param charVarcharStandardSemanticsOverride Analysis-time standardSemantics for from_avro;
+ * `None` reads the live session conf (file scans).
*/
private[sql] class AvroDeserializer(
rootAvroType: Schema,
@@ -62,7 +70,9 @@ private[sql] class AvroDeserializer(
useStableIdForUnionType: Boolean,
stableIdPrefixForUnionType: String,
recursiveFieldMaxDepth: Int,
- dataSchema: Option[StructType]) {
+ dataSchema: Option[StructType],
+ applyCharVarcharAssignment: Boolean = false,
+ charVarcharStandardSemanticsOverride: Option[Boolean] = None) {
def this(
rootAvroType: Schema,
@@ -83,6 +93,29 @@ private[sql] class AvroDeserializer(
dataSchema = None)
}
+ def this(
+ rootAvroType: Schema,
+ rootCatalystType: DataType,
+ datetimeRebaseMode: String,
+ useStableIdForUnionType: Boolean,
+ stableIdPrefixForUnionType: String,
+ recursiveFieldMaxDepth: Int,
+ applyCharVarcharAssignment: Boolean,
+ charVarcharStandardSemanticsOverride: Option[Boolean]) = {
+ this(
+ rootAvroType,
+ rootCatalystType,
+ positionalFieldMatch = false,
+ RebaseSpec(LegacyBehaviorPolicy.withName(datetimeRebaseMode)),
+ new NoopFilters,
+ useStableIdForUnionType,
+ stableIdPrefixForUnionType,
+ recursiveFieldMaxDepth,
+ dataSchema = None,
+ applyCharVarcharAssignment,
+ charVarcharStandardSemanticsOverride)
+ }
+
private lazy val decimalConversions = new DecimalConversion()
private val dateRebaseFunc = DataSourceUtils.createDateRebaseFuncInRead(
@@ -91,6 +124,10 @@ private[sql] class AvroDeserializer(
private val timestampRebaseFunc = DataSourceUtils.createTimestampRebaseFuncInRead(
datetimeRebaseSpec, "Avro")
+ // Lazy so converter setup cannot observe a default false if this field moves (JacksonParser).
+ private lazy val charVarcharStandardSemantics =
+ charVarcharStandardSemanticsOverride.getOrElse(SQLConf.get.charVarcharStandardSemantics)
+
private val converter: Any => Option[Any] = try {
rootCatalystType match {
// A shortcut for empty schema.
@@ -260,7 +297,7 @@ private[sql] class AvroDeserializer(
case (DOUBLE, DoubleType) => (updater, ordinal, value) =>
updater.setDouble(ordinal, value.asInstanceOf[Double])
- case (STRING, _: StringType) => (updater, ordinal, value) =>
+ case (STRING, st: StringType) => (updater, ordinal, value) =>
val str = value match {
case s: String => UTF8String.fromString(s)
case s: Utf8 =>
@@ -268,7 +305,14 @@ private[sql] class AvroDeserializer(
System.arraycopy(s.getBytes, 0, bytes, 0, s.getByteLength)
UTF8String.fromBytes(bytes)
}
- updater.set(ordinal, str)
+ // File scans leave CHAR padding to ApplyCharTypePadding. from_avro assigns STRING
+ // values only; ENUM symbols stay as-is so a padded CHAR cannot invent a symbol.
+ val assigned = if (applyCharVarcharAssignment) {
+ CharVarcharUtils.applyTextParseSemantics(str, st)
+ } else {
+ str
+ }
+ updater.set(ordinal, assigned)
case (ENUM, _: StringType) => (updater, ordinal, value) =>
updater.set(ordinal, UTF8String.fromString(value.toString))
@@ -356,10 +400,13 @@ private[sql] class AvroDeserializer(
updater.set(ordinal, result)
case (MAP, MapType(keyType: StringType, valueType, valueContainsNull)) =>
- val keyWriter = newWriter(SchemaBuilder.builder().stringType(), keyType,
+ // Decode keys as unbounded STRING so CHAR/VARCHAR assignment never rewrites them.
+ // Identity length-check matches JSON/XML map keys (no pad/trim).
+ val keyWriter = newWriter(SchemaBuilder.builder().stringType(), StringType,
avroPath :+ "key", catalystPath :+ "key")
val valueWriter = newWriter(avroType.getValueType, valueType,
avroPath :+ "value", catalystPath :+ "value")
+ val mapKeyChecker = makeAvroMapKeyChecker(keyType)
(updater, ordinal, value) =>
val map = value.asInstanceOf[java.util.Map[AnyRef, AnyRef]]
val keyArray = createArrayData(keyType, map.size())
@@ -372,6 +419,7 @@ private[sql] class AvroDeserializer(
val entry = iter.next()
assert(entry.getKey != null)
keyWriter(keyUpdater, i, entry.getKey)
+ mapKeyChecker.foreach(_(keyArray.getUTF8String(i)))
if (entry.getValue == null) {
if (!valueContainsNull) {
throw new SparkRuntimeException(
@@ -530,6 +578,28 @@ private[sql] class AvroDeserializer(
}
}
+ /**
+ * Length-check CHAR/VARCHAR Avro map keys without pad/trim. Built once per map converter.
+ * Gated on standardSemantics only (not from_avro assignment): scans must reject off-width
+ * keys too, otherwise ApplyCharTypePadding can rewrite CHAR keys and HashMap.put last-wins.
+ * UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY lets from_avro PERMISSIVE swallow the whole record.
+ */
+ private def makeAvroMapKeyChecker(keyType: DataType): Option[UTF8String => Unit] = {
+ if (!charVarcharStandardSemantics) {
+ None
+ } else {
+ keyType match {
+ case _: CharType | _: VarcharType =>
+ Some { key =>
+ if (CharVarcharUtils.charVarcharMapKeyViolatesLength(key, keyType)) {
+ throw QueryExecutionErrors.unsupportedAvroCharVarcharMapKey(key, keyType)
+ }
+ }
+ case _ => None
+ }
+ }
+ }
+
private def createArrayData(elementType: DataType, length: Int): ArrayData = elementType match {
case BooleanType => UnsafeArrayData.fromPrimitiveArray(new Array[Boolean](length))
case ByteType => UnsafeArrayData.fromPrimitiveArray(new Array[Byte](length))
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
index fa091a8d4b32..19f94d2f6b84 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
@@ -2234,6 +2234,7 @@ class BasicCharVarcharTestSuite extends SharedSparkSession {
withTempPath { file =>
val ser = new org.apache.spark.sql.avro.AvroSerializer(
nestedSchema, nestedAvro, nullable = false)
+ // File-scan deserializers do not assign CHAR padding; values are stored padded.
val row = org.apache.spark.sql.catalyst.InternalRow(
org.apache.spark.unsafe.types.UTF8String.fromString("ab "),
org.apache.spark.sql.catalyst.InternalRow(
@@ -2272,6 +2273,73 @@ class BasicCharVarcharTestSuite extends SharedSparkSession {
}
}
+ // SPARK-60242: from_avro-equivalent decode assigns CHAR/VARCHAR STRING values.
+ // File-scan deserializers (default) and to_avro encoding do not.
+ Seq(CharType(4), VarcharType(4)).foreach { dt =>
+ val avro = converters.toAvroType(dt, nullable = false)
+ val fromAvroDeser = new org.apache.spark.sql.avro.AvroDeserializer(
+ avro, dt, "CORRECTED", false, "", -1,
+ applyCharVarcharAssignment = true,
+ charVarcharStandardSemanticsOverride = None)
+ val scanDeser = new org.apache.spark.sql.avro.AvroDeserializer(
+ avro, dt, "CORRECTED", false, "", -1)
+ val padded = dt match {
+ case _: CharType => "ab "
+ case _ => "ab"
+ }
+ assert(
+ fromAvroDeser.deserialize(new org.apache.avro.util.Utf8("ab")).get.toString === padded,
+ s"from_avro-equivalent decode should assign $dt")
+ assert(
+ scanDeser.deserialize(new org.apache.avro.util.Utf8("ab")).get.toString === "ab",
+ s"file-scan decode should not assign $dt")
+ checkError(
+ exception = intercept[SparkRuntimeException] {
+ fromAvroDeser.deserialize(new org.apache.avro.util.Utf8("abcde"))
+ },
+ condition = "EXCEED_LIMIT_LENGTH",
+ parameters = Map("limit" -> "4"))
+ val ser = new org.apache.spark.sql.avro.AvroSerializer(dt, avro, nullable = false)
+ val encoded = ser.serialize(
+ org.apache.spark.unsafe.types.UTF8String.fromString("ab"))
+ .asInstanceOf[org.apache.avro.util.Utf8]
+ assert(encoded.toString === "ab", s"to_avro-equivalent encode should not assign $dt")
+ }
+
+ val enumSchema = org.apache.avro.Schema.createEnum(
+ "Spark60242Enum", "", "", java.util.Arrays.asList("ab", "cd"))
+ val enumVal = new org.apache.avro.generic.GenericData.EnumSymbol(enumSchema, "ab")
+ val enumDeser = new org.apache.spark.sql.avro.AvroDeserializer(
+ enumSchema, CharType(4), "CORRECTED", false, "", -1,
+ applyCharVarcharAssignment = true,
+ charVarcharStandardSemanticsOverride = None)
+ assert(
+ enumDeser.deserialize(enumVal).get.toString === "ab",
+ "ENUM symbols must not be CHAR-padded")
+
+ val mapDt = MapType(CharType(2), IntegerType)
+ val mapAvro = converters.toAvroType(mapDt, nullable = false)
+ val shortKeyMap = new java.util.HashMap[AnyRef, AnyRef]()
+ shortKeyMap.put(new org.apache.avro.util.Utf8("a"), Integer.valueOf(1))
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "false") {
+ val captured = new org.apache.spark.sql.avro.AvroDeserializer(
+ mapAvro, mapDt, "CORRECTED", false, "", -1,
+ applyCharVarcharAssignment = false,
+ charVarcharStandardSemanticsOverride = Some(true))
+ checkError(
+ exception = intercept[SparkRuntimeException] {
+ captured.deserialize(shortKeyMap)
+ },
+ condition = "UNSUPPORTED_AVRO_CHAR_VARCHAR_MAP_KEY",
+ sqlState = "0A000",
+ parameters = Map("key" -> "'a'", "dataType" -> "\"CHAR(2)\""))
+ val live = new org.apache.spark.sql.avro.AvroDeserializer(
+ mapAvro, mapDt, "CORRECTED", false, "", -1)
+ val liveMap = live.deserialize(shortKeyMap).get
+ .asInstanceOf[org.apache.spark.sql.catalyst.util.MapData]
+ assert(liveMap.keyArray().getUTF8String(0).toString === "a")
+ }
+
// JSON has no embedded schema, so a user-specified schema supplies the logical type.
withTempPath { dir =>
val path = dir.getCanonicalPath
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org