You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "srielau (via GitHub)" <gi...@apache.org> on 2026/09/21 05:17:20 UTC
[PR] [WIP][SQL][HIVE] Support first-class CHAR/VARCHAR in script TRANSFORM output [spark]
srielau opened a new pull request, #58936:
URL: https://github.com/apache/spark/pull/58936
### What changes were proposed in this pull request?
Support first-class CHAR/VARCHAR output from script TRANSFORM when first-class semantics are enabled. Split out from [#58584](https://github.com/apache/spark/pull/58584) (Hive inspector conversion + UDF/UDAF/UDTF) per review feedback.
- **No-SerDe path:** CHAR/VARCHAR scalar, nested, and complex output with padding, overflow (`EXCEED_LIMIT_LENGTH`), null-token, and malformed-field handling.
- **No-SerDe nested maps:** Restore JSON string keys to the declared physical key type through a per-row `ArrayBasedMapBuilder` that validates null and duplicate converted keys.
- **SerDe path:** Rewrite `LazySimpleSerDe` CHAR/VARCHAR output types to STRING so Spark applies first-class length checks. Non-`LazySimpleSerDe` SerDes keep the declared CHAR/VARCHAR schema.
- **Legacy:** Legacy flag-off behavior remains unchanged.
This PR depends on [#58584](https://github.com/apache/spark/pull/58584) for Hive inspector CHAR/VARCHAR conversion. A JIRA subtask under SPARK-59277 will be created for this work.
### Why are the changes needed?
With `spark.sql.charVarchar.standardSemantics.enabled=true`, script TRANSFORM output was not aware of CHAR/VARCHAR types. Bounded-string values passed through without padding or length enforcement, and nested output containing non-string map keys was not restored from the JSON string representation.
### Does this PR introduce _any_ user-facing change?
Yes. When first-class CHAR/VARCHAR semantics are enabled, script TRANSFORM now pads CHAR output, enforces length limits on both CHAR and VARCHAR, and correctly handles nested CHAR/VARCHAR in arrays, maps, and structs. Legacy flag-off behavior remains unchanged.
### How was this patch tested?
Added focused coverage for:
- Scalar CHAR/VARCHAR output without SerDe (padding, overflow).
- Nested CHAR/VARCHAR (arrays, maps with non-string keys, structs) without SerDe.
- Malformed JSON and colliding map-key restoration without SerDe.
- Colliding row followed by valid row in the same partition (builder-leak regression).
- Nested collated CHAR/VARCHAR with Hive SerDe.
- Scalar CHAR/VARCHAR overflow with Hive SerDe.
- `LazySimpleSerDe`-only CHAR/VARCHAR type rewrite verification.
Ran:
```
sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 \
-Dsbt.override.build.repos=true \
'sql/testOnly org.apache.spark.sql.execution.SparkScriptTransformationSuite -- -z SPARK-59277'
sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 \
-Dsbt.override.build.repos=true \
'hive/testOnly org.apache.spark.sql.hive.execution.HiveScriptTransformationSuite -- -z SPARK-59277'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Auto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59683][SQL][HIVE] Support first-class CHAR/VARCHAR in script TRANSFORM output [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #58936:
URL: https://github.com/apache/spark/pull/58936#discussion_r4168977689
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -241,6 +276,29 @@ trait BaseScriptTransformationExec extends UnaryExecNode {
data => IntervalUtils.microsToDuration(
IntervalUtils.castStringToDTInterval(UTF8String.fromString(data), start, end)),
converter)
+ case dt @ (_: ArrayType | _: MapType | _: StructType)
+ if standardCharVarcharSemantics && CharVarcharUtils.hasCharVarchar(dt) =>
+ val physicalType = ScriptTransformationIOSchema.toUnboundedStringType(dt)
+ // JSON object keys are strings. Cast them to the declared map key type after parsing.
+ val jsonType = ScriptTransformationIOSchema.toJsonMapKeyType(physicalType)
+ val complexTypeFactory = JsonToStructs(
+ jsonType,
+ ioschema.outputSerdeProps.toMap,
+ Literal(null),
+ Some(conf.sessionLocalTimeZone))
+ val parsedToPhysical = if (jsonType.sameType(physicalType)) {
+ identity[Any] _
+ } else {
+ val restoreMapKeys = ScriptTransformationIOSchema.makeJsonMapKeyRestorer(
+ physicalType, Some(conf.sessionLocalTimeZone))
+ value: Any => restoreMapKeys(value)
+ }
+ val toScala = CatalystTypeConverters.createToScalaConverter(physicalType)
+ val parser = wrapperConvertException(
+ data => parsedToPhysical(
+ complexTypeFactory.nullSafeEval(UTF8String.fromString(data))),
+ identity)
+ data => converter(toScala(parser(data)))
Review Comment:
Addressed. Nested conversion now stays in Catalyst: JSON parse and non-string key restore remain inside the malformed-field wrapper, then `makeJsonToTargetRestorer` applies CHAR/VARCHAR write-side checks and rebuilds every map with a fresh `ArrayBasedMapBuilder`. Distinct JSON keys that pad to the same CHAR/VARCHAR key now raise `DUPLICATED_MAP_KEY` under the default EXCEPTION policy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59683][SQL][HIVE] Support first-class CHAR/VARCHAR in script TRANSFORM output [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #58936:
URL: https://github.com/apache/spark/pull/58936#discussion_r4168980201
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -62,6 +79,10 @@ trait BaseScriptTransformationExec extends UnaryExecNode {
override def outputPartitioning: Partitioning = child.outputPartitioning
+ // Snapshot on the driver. SparkPlan.session is @transient, so executor `conf` is the
Review Comment:
Updated. The snapshot is for PERSISTED `CHAR_VARCHAR_STANDARD_SEMANTICS` at plan construction, not because the executor falls back to a default SQLConf. Task `SQLConf.get` is the executing session's ReadOnlySQLConf; `SparkPlan.session` is `@transient`, so executor `conf` cannot recover the planning session.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -376,6 +434,98 @@ case class ScriptTransformationIOSchema(
}
object ScriptTransformationIOSchema {
+ private[sql] def toUnboundedStringType(dataType: DataType): DataType = {
+ dataType.transformRecursively {
+ case c: CharType => c.toStringType
+ case v: VarcharType => v.toStringType
+ }
+ }
+
+ // JSON object keys are always strings. Rewrite every map key, including nested maps.
+ // `transformRecursively` would stop at the first matching MapType and skip children.
+ private[sql] def toJsonMapKeyType(dataType: DataType): DataType = dataType match {
+ case ArrayType(et, n) => ArrayType(toJsonMapKeyType(et), n)
+ case MapType(kt, vt, n) =>
+ val jsonKey = if (kt.isInstanceOf[StringType]) kt else StringType
+ MapType(jsonKey, toJsonMapKeyType(vt), n)
+ case StructType(fields) =>
+ StructType(fields.map(f => f.copy(dataType = toJsonMapKeyType(f.dataType))))
+ case other => other
+ }
+
+ /**
+ * Build a per-call map-key restorer that converts parsed JSON string keys
+ * back to the declared physical key type and validates the result through a
+ * fresh [[ArrayBasedMapBuilder]] on every invocation, so a failed key
+ * conversion or duplicate key cannot leave shared state dirty for the next row.
+ */
+ private[sql] def makeJsonMapKeyRestorer(
+ targetType: DataType,
+ timeZoneId: Option[String]): Any => Any = {
+ val jsonType = toJsonMapKeyType(targetType)
+
+ def make(jt: DataType, tt: DataType): Any => Any = (jt, tt) match {
+ case (ArrayType(jet, _), ArrayType(tet, _)) =>
+ val elem = make(jet, tet)
+ (input: Any) => {
+ val arr = input.asInstanceOf[ArrayData]
+ val n = arr.numElements()
+ val out = new Array[Any](n)
+ var i = 0
+ while (i < n) {
+ out(i) = if (arr.isNullAt(i)) null
Review Comment:
Added fixtures for null array elements, null map values, and null struct fields in the nested CHAR/VARCHAR conversion test.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -201,7 +222,21 @@ trait BaseScriptTransformationExec extends UnaryExecNode {
private lazy val outputFieldWriters: Seq[String => Any] = output.map { attr =>
val converter = CatalystTypeConverters.createToCatalystConverter(attr.dataType)
attr.dataType match {
- case StringType => wrapperConvertException(data => data, converter)
+ case dt @ (_: CharType | _: VarcharType) =>
+ // Preserve-only keeps CHAR/VARCHAR in the schema but does not apply SQL pad/overflow.
Review Comment:
Reworded. The comment now only states that no-SerDe scalar CHAR/VARCHAR stays unsupported without standard semantics, and why that case must precede `StringType`.
##########
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala:
##########
@@ -372,6 +380,163 @@ class HiveScriptTransformationSuite extends BaseScriptTransformationSuite with T
}
}
+ test("SPARK-59277: TRANSFORM supports nested collated CHAR/VARCHAR with Hive SerDe") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ val query = sql(
+ """
+ |SELECT TRANSFORM(
+ | array(CAST('ab' AS CHAR(4) COLLATE UTF8_LCASE)),
+ | named_struct('value', CAST('xyz' AS VARCHAR(6) COLLATE UNICODE_CI)))
+ |USING 'cat'
+ |AS (
+ | chars ARRAY<CHAR(4) COLLATE UTF8_LCASE>,
+ | nested STRUCT<value: VARCHAR(6) COLLATE UNICODE_CI>)
+ |FROM VALUES (1) input(dummy)
+ |""".stripMargin)
+ assert(query.schema.map(_.dataType) === Seq(
+ ArrayType(CharType(4, "UTF8_LCASE")),
+ StructType(Seq(StructField("value", VarcharType(6, "UNICODE_CI"))))))
+ checkAnswer(query, Row(Seq("ab "), Row("xyz")))
+ }
+ }
+
+ test("SPARK-59277: TRANSFORM CHAR overflow with Hive SerDe raises EXCEED_LIMIT_LENGTH") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ val exception = intercept[Exception] {
+ sql(
+ """
+ |SELECT TRANSFORM('abcdef')
+ |USING 'cat'
+ |AS (c CHAR(4))
+ |FROM VALUES (1) input(dummy)
+ |""".stripMargin).collect()
+ }
+ val runtimeException = exception match {
+ case s: org.apache.spark.SparkRuntimeException => s
+ case other =>
+ other.getCause.asInstanceOf[org.apache.spark.SparkRuntimeException]
+ }
+ checkError(
+ exception = runtimeException,
+ condition = "EXCEED_LIMIT_LENGTH",
+ parameters = Map("limit" -> "4"))
+ }
+ }
+
+ test("SPARK-59277: TRANSFORM VARCHAR overflow with Hive SerDe raises EXCEED_LIMIT_LENGTH") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ val exception = intercept[Exception] {
+ sql(
+ """
+ |SELECT TRANSFORM('abcdefgh')
+ |USING 'cat'
+ |AS (v VARCHAR(5))
+ |FROM VALUES (1) input(dummy)
+ |""".stripMargin).collect()
+ }
+ val runtimeException = exception match {
+ case s: org.apache.spark.SparkRuntimeException => s
+ case other =>
+ other.getCause.asInstanceOf[org.apache.spark.SparkRuntimeException]
+ }
+ checkError(
+ exception = runtimeException,
+ condition = "EXCEED_LIMIT_LENGTH",
+ parameters = Map("limit" -> "5"))
+ }
+ }
+
+ test("SPARK-59277: nested CHAR/VARCHAR overflow with Hive SerDe") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ Seq(
+ """
+ |SELECT TRANSFORM(array('abcdef'))
+ |USING 'cat'
+ |AS (value ARRAY<CHAR(4)>)
+ |FROM VALUES (1) input(dummy)
+ |""".stripMargin,
+ """
+ |SELECT TRANSFORM(named_struct('value', 'abcdef'))
+ |USING 'cat'
+ |AS (value STRUCT<value: VARCHAR(4)>)
+ |FROM VALUES (1) input(dummy)
+ |""".stripMargin).foreach { query =>
+ val exception = intercept[Exception] {
+ sql(query).collect()
+ }
+ val runtimeException = exception match {
+ case s: org.apache.spark.SparkRuntimeException => s
+ case other =>
+ other.getCause.asInstanceOf[org.apache.spark.SparkRuntimeException]
+ }
+ checkError(
+ exception = runtimeException,
+ condition = "EXCEED_LIMIT_LENGTH",
+ parameters = Map("limit" -> "4"))
+ }
+ }
+ }
+
+ test("SPARK-59277: output SerDe CHAR/VARCHAR rewrite is LazySimpleSerDe-only") {
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ val output = Seq(
+ AttributeReference("c", CharType(4))(),
+ AttributeReference("v", VarcharType(5))(),
+ AttributeReference("nested", ArrayType(CharType(4)))())
+
+ val (_, lazySoi) = HiveScriptIOSchema.initOutputSerDe(hiveIOSchema, output).get
+ assert(lazySoi.getAllStructFieldRefs.get(0).getFieldObjectInspector.getTypeName ===
+ "string")
+ assert(lazySoi.getAllStructFieldRefs.get(1).getFieldObjectInspector.getTypeName ===
+ "string")
+ assert(lazySoi.getAllStructFieldRefs.get(2).getFieldObjectInspector.getTypeName ===
+ "array<string>")
+
+ val subclassSchema = hiveIOSchema.copy(
+ outputSerdeClass = Some(classOf[TestLazySimpleSerDe].getCanonicalName))
+ val (_, subclassSoi) = HiveScriptIOSchema.initOutputSerDe(subclassSchema, output).get
+ assert(subclassSoi.getAllStructFieldRefs.get(0).getFieldObjectInspector.getTypeName ===
+ "string")
+
+ SchemaCapturingSerDe.lastColumnTypes = null
+ val customSchema = defaultIOSchema.copy(
+ outputSerdeClass = Some(classOf[SchemaCapturingSerDe].getCanonicalName))
+ HiveScriptIOSchema.initOutputSerDe(customSchema, output)
+ val captured = SchemaCapturingSerDe.lastColumnTypes
+ assert(captured.contains("char(4)"))
+ assert(captured.contains("varchar(5)"))
+ assert(captured.contains("array<char(4)>"))
+ }
+ }
+
+ test("SPARK-59683: preserve-only does not rewrite LazySimpleSerDe CHAR/VARCHAR") {
+ withSQLConf(
+ SQLConf.PRESERVE_CHAR_VARCHAR_TYPE_INFO.key -> "true",
+ SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "false") {
+ val output = Seq(
+ AttributeReference("c", CharType(4))(),
+ AttributeReference("v", VarcharType(5))(),
+ AttributeReference("nested", ArrayType(CharType(4)))())
+ scala.util.Try(HiveScriptIOSchema.initOutputSerDe(hiveIOSchema, output).get) match {
+ case scala.util.Success((_, soi)) =>
+ val typeName = soi.getAllStructFieldRefs.get(0).getFieldObjectInspector.getTypeName
+ assert(typeName === "char(4)",
+ s"preserve-only must not rewrite CHAR to STRING, found $typeName")
+ assert(soi.getAllStructFieldRefs.get(1).getFieldObjectInspector.getTypeName ===
+ "varchar(5)")
+ assert(soi.getAllStructFieldRefs.get(2).getFieldObjectInspector.getTypeName ===
+ "array<char(4)>")
+ case scala.util.Failure(_) =>
Review Comment:
Removed the catch-all Failure. The test now asserts LazySimpleSerDe inspectors stay `char(4)` / `varchar(5)` / `array<char(4)>` under preserve-only.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -201,7 +222,21 @@ trait BaseScriptTransformationExec extends UnaryExecNode {
private lazy val outputFieldWriters: Seq[String => Any] = output.map { attr =>
val converter = CatalystTypeConverters.createToCatalystConverter(attr.dataType)
attr.dataType match {
- case StringType => wrapperConvertException(data => data, converter)
+ case dt @ (_: CharType | _: VarcharType) =>
+ // Preserve-only keeps CHAR/VARCHAR in the schema but does not apply SQL pad/overflow.
+ // Match CHAR/VARCHAR before `_: StringType` (they extend StringType).
+ if (!standardCharVarcharSemantics) {
+ throw QueryExecutionErrors.outputDataTypeUnsupportedByNodeWithoutSerdeError(
+ nodeName, dt)
+ }
+ // Do not use the SerDe null-on-error wrapper; that would hide EXCEED_LIMIT_LENGTH.
+ (data: String) =>
+ if (data == ioschema.outputRowFormatMap("TOK_TABLEROWFORMATNULL")) {
Review Comment:
Added a scalar CHAR/VARCHAR null-token test (null input / configured null token) alongside the padding cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59683][SQL][HIVE] Support first-class CHAR/VARCHAR in script TRANSFORM output [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on PR #58936:
URL: https://github.com/apache/spark/pull/58936#issuecomment-5959630840
Addressed the review summary in 7f80eb944c9.
Nested CHAR/VARCHAR conversion no longer round-trips through Scala / CatalystTypeConverters. Maps always rebuild through ArrayBasedMapBuilder so CHAR/VARCHAR key collisions follow MAP_KEY_DEDUP_POLICY, and EXCEED_LIMIT_LENGTH stays outside the malformed-field wrapper. Inline comments are replied on the threads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59683][SQL][HIVE] Support first-class CHAR/VARCHAR in script TRANSFORM output [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58936:
URL: https://github.com/apache/spark/pull/58936#discussion_r4164780396
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -241,6 +276,29 @@ trait BaseScriptTransformationExec extends UnaryExecNode {
data => IntervalUtils.microsToDuration(
IntervalUtils.castStringToDTInterval(UTF8String.fromString(data), start, end)),
converter)
+ case dt @ (_: ArrayType | _: MapType | _: StructType)
+ if standardCharVarcharSemantics && CharVarcharUtils.hasCharVarchar(dt) =>
+ val physicalType = ScriptTransformationIOSchema.toUnboundedStringType(dt)
+ // JSON object keys are strings. Cast them to the declared map key type after parsing.
+ val jsonType = ScriptTransformationIOSchema.toJsonMapKeyType(physicalType)
+ val complexTypeFactory = JsonToStructs(
+ jsonType,
+ ioschema.outputSerdeProps.toMap,
+ Literal(null),
+ Some(conf.sessionLocalTimeZone))
+ val parsedToPhysical = if (jsonType.sameType(physicalType)) {
+ identity[Any] _
+ } else {
+ val restoreMapKeys = ScriptTransformationIOSchema.makeJsonMapKeyRestorer(
+ physicalType, Some(conf.sessionLocalTimeZone))
+ value: Any => restoreMapKeys(value)
+ }
+ val toScala = CatalystTypeConverters.createToScalaConverter(physicalType)
+ val parser = wrapperConvertException(
+ data => parsedToPhysical(
+ complexTypeFactory.nullSafeEval(UTF8String.fromString(data))),
+ identity)
+ data => converter(toScala(parser(data)))
Review Comment:
**Non-blocking (P2):** For standard-semantics no-SerDe output, CHAR/VARCHAR map keys have already been erased to STRING here, so distinct JSON keys such as `a` and `a ` can both normalize to the same padded key during the final conversion. `CatalystTypeConverters` then constructs `ArrayBasedMapData` without duplicate validation, which can emit an invalid map instead of raising `DUPLICATED_MAP_KEY` or applying `LAST_WIN`.
See **Shared repair plan 1** in the review body.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -201,7 +222,21 @@ trait BaseScriptTransformationExec extends UnaryExecNode {
private lazy val outputFieldWriters: Seq[String => Any] = output.map { attr =>
val converter = CatalystTypeConverters.createToCatalystConverter(attr.dataType)
attr.dataType match {
- case StringType => wrapperConvertException(data => data, converter)
+ case dt @ (_: CharType | _: VarcharType) =>
+ // Preserve-only keeps CHAR/VARCHAR in the schema but does not apply SQL pad/overflow.
Review Comment:
**Nit (P3):** Preserve-only mode is not defined by omitting CHAR/VARCHAR padding or overflow checks: the Catalyst converters apply the bounded-string write checks for these types. This branch keeps no-SerDe scalar output unsupported when standard semantics are disabled, so the comment should state that compatibility boundary without claiming different converter semantics.
**Verification:**
- **Inspection:** Compare the final wording with CharConverter/VarcharConverter and preserve-only encoder behavior and confirm it states no broader semantic claim.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -62,6 +79,10 @@ trait BaseScriptTransformationExec extends UnaryExecNode {
override def outputPartitioning: Partitioning = child.outputPartitioning
+ // Snapshot on the driver. SparkPlan.session is @transient, so executor `conf` is the
Review Comment:
**Nit (P3):** This rationale is inaccurate: when the transient session is unavailable in a running task, `SQLConf.get` resolves to the task's `ReadOnlySQLConf`, which reads the driver-propagated query properties rather than a default SQLConf. Please describe the actual reason this value needs to be snapshotted, if it still does.
**Verification:**
- **Inspection:** Trace the final wording through SparkPlan.conf, SQLConfHelper.conf, and running-task SQLConf.get behavior and confirm every factual claim matches those paths.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -376,6 +434,98 @@ case class ScriptTransformationIOSchema(
}
object ScriptTransformationIOSchema {
+ private[sql] def toUnboundedStringType(dataType: DataType): DataType = {
+ dataType.transformRecursively {
+ case c: CharType => c.toStringType
+ case v: VarcharType => v.toStringType
+ }
+ }
+
+ // JSON object keys are always strings. Rewrite every map key, including nested maps.
+ // `transformRecursively` would stop at the first matching MapType and skip children.
+ private[sql] def toJsonMapKeyType(dataType: DataType): DataType = dataType match {
+ case ArrayType(et, n) => ArrayType(toJsonMapKeyType(et), n)
+ case MapType(kt, vt, n) =>
+ val jsonKey = if (kt.isInstanceOf[StringType]) kt else StringType
+ MapType(jsonKey, toJsonMapKeyType(vt), n)
+ case StructType(fields) =>
+ StructType(fields.map(f => f.copy(dataType = toJsonMapKeyType(f.dataType))))
+ case other => other
+ }
+
+ /**
+ * Build a per-call map-key restorer that converts parsed JSON string keys
+ * back to the declared physical key type and validates the result through a
+ * fresh [[ArrayBasedMapBuilder]] on every invocation, so a failed key
+ * conversion or duplicate key cannot leave shared state dirty for the next row.
+ */
+ private[sql] def makeJsonMapKeyRestorer(
+ targetType: DataType,
+ timeZoneId: Option[String]): Any => Any = {
+ val jsonType = toJsonMapKeyType(targetType)
+
+ def make(jt: DataType, tt: DataType): Any => Any = (jt, tt) match {
+ case (ArrayType(jet, _), ArrayType(tet, _)) =>
+ val elem = make(jet, tet)
+ (input: Any) => {
+ val arr = input.asInstanceOf[ArrayData]
+ val n = arr.numElements()
+ val out = new Array[Any](n)
+ var i = 0
+ while (i < n) {
+ out(i) = if (arr.isNullAt(i)) null
Review Comment:
**Non-blocking (P2):** The recursive no-SerDe restorer has separate null-preservation branches for array elements, map values, and struct fields, but the new valid nested fixtures exercise only non-null members. A regression here could be swallowed by `wrapperConvertException` and turn the entire output field into null, so the tests need valid nested-null cases that assert the null remains at its original nesting level.
See **Shared repair plan 1** in the review body.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -201,7 +222,21 @@ trait BaseScriptTransformationExec extends UnaryExecNode {
private lazy val outputFieldWriters: Seq[String => Any] = output.map { attr =>
val converter = CatalystTypeConverters.createToCatalystConverter(attr.dataType)
attr.dataType match {
- case StringType => wrapperConvertException(data => data, converter)
+ case dt @ (_: CharType | _: VarcharType) =>
+ // Preserve-only keeps CHAR/VARCHAR in the schema but does not apply SQL pad/overflow.
+ // Match CHAR/VARCHAR before `_: StringType` (they extend StringType).
+ if (!standardCharVarcharSemantics) {
+ throw QueryExecutionErrors.outputDataTypeUnsupportedByNodeWithoutSerdeError(
+ nodeName, dt)
+ }
+ // Do not use the SerDe null-on-error wrapper; that would hide EXCEED_LIMIT_LENGTH.
+ (data: String) =>
+ if (data == ioschema.outputRowFormatMap("TOK_TABLEROWFORMATNULL")) {
Review Comment:
**Non-blocking (P2):** This is a new null-token path specific to standard-semantics scalar CHAR/VARCHAR output, while the added scalar tests cover only padding and overflow. Please add a configured null-token case that asserts a SQL null result; otherwise a regression can turn the marker into padded literal data or an overflow without affecting the existing STRING null tests.
**Verification:**
- **Regression:** Exercise scalar CHAR/VARCHAR output with the configured null token and assert a null Catalyst/SQL result under standard semantics.
##########
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala:
##########
@@ -372,6 +380,163 @@ class HiveScriptTransformationSuite extends BaseScriptTransformationSuite with T
}
}
+ test("SPARK-59277: TRANSFORM supports nested collated CHAR/VARCHAR with Hive SerDe") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ val query = sql(
+ """
+ |SELECT TRANSFORM(
+ | array(CAST('ab' AS CHAR(4) COLLATE UTF8_LCASE)),
+ | named_struct('value', CAST('xyz' AS VARCHAR(6) COLLATE UNICODE_CI)))
+ |USING 'cat'
+ |AS (
+ | chars ARRAY<CHAR(4) COLLATE UTF8_LCASE>,
+ | nested STRUCT<value: VARCHAR(6) COLLATE UNICODE_CI>)
+ |FROM VALUES (1) input(dummy)
+ |""".stripMargin)
+ assert(query.schema.map(_.dataType) === Seq(
+ ArrayType(CharType(4, "UTF8_LCASE")),
+ StructType(Seq(StructField("value", VarcharType(6, "UNICODE_CI"))))))
+ checkAnswer(query, Row(Seq("ab "), Row("xyz")))
+ }
+ }
+
+ test("SPARK-59277: TRANSFORM CHAR overflow with Hive SerDe raises EXCEED_LIMIT_LENGTH") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ val exception = intercept[Exception] {
+ sql(
+ """
+ |SELECT TRANSFORM('abcdef')
+ |USING 'cat'
+ |AS (c CHAR(4))
+ |FROM VALUES (1) input(dummy)
+ |""".stripMargin).collect()
+ }
+ val runtimeException = exception match {
+ case s: org.apache.spark.SparkRuntimeException => s
+ case other =>
+ other.getCause.asInstanceOf[org.apache.spark.SparkRuntimeException]
+ }
+ checkError(
+ exception = runtimeException,
+ condition = "EXCEED_LIMIT_LENGTH",
+ parameters = Map("limit" -> "4"))
+ }
+ }
+
+ test("SPARK-59277: TRANSFORM VARCHAR overflow with Hive SerDe raises EXCEED_LIMIT_LENGTH") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ val exception = intercept[Exception] {
+ sql(
+ """
+ |SELECT TRANSFORM('abcdefgh')
+ |USING 'cat'
+ |AS (v VARCHAR(5))
+ |FROM VALUES (1) input(dummy)
+ |""".stripMargin).collect()
+ }
+ val runtimeException = exception match {
+ case s: org.apache.spark.SparkRuntimeException => s
+ case other =>
+ other.getCause.asInstanceOf[org.apache.spark.SparkRuntimeException]
+ }
+ checkError(
+ exception = runtimeException,
+ condition = "EXCEED_LIMIT_LENGTH",
+ parameters = Map("limit" -> "5"))
+ }
+ }
+
+ test("SPARK-59277: nested CHAR/VARCHAR overflow with Hive SerDe") {
+ assume(TestUtils.testCommandAvailable("/bin/bash"))
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ Seq(
+ """
+ |SELECT TRANSFORM(array('abcdef'))
+ |USING 'cat'
+ |AS (value ARRAY<CHAR(4)>)
+ |FROM VALUES (1) input(dummy)
+ |""".stripMargin,
+ """
+ |SELECT TRANSFORM(named_struct('value', 'abcdef'))
+ |USING 'cat'
+ |AS (value STRUCT<value: VARCHAR(4)>)
+ |FROM VALUES (1) input(dummy)
+ |""".stripMargin).foreach { query =>
+ val exception = intercept[Exception] {
+ sql(query).collect()
+ }
+ val runtimeException = exception match {
+ case s: org.apache.spark.SparkRuntimeException => s
+ case other =>
+ other.getCause.asInstanceOf[org.apache.spark.SparkRuntimeException]
+ }
+ checkError(
+ exception = runtimeException,
+ condition = "EXCEED_LIMIT_LENGTH",
+ parameters = Map("limit" -> "4"))
+ }
+ }
+ }
+
+ test("SPARK-59277: output SerDe CHAR/VARCHAR rewrite is LazySimpleSerDe-only") {
+ withSQLConf(SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "true") {
+ val output = Seq(
+ AttributeReference("c", CharType(4))(),
+ AttributeReference("v", VarcharType(5))(),
+ AttributeReference("nested", ArrayType(CharType(4)))())
+
+ val (_, lazySoi) = HiveScriptIOSchema.initOutputSerDe(hiveIOSchema, output).get
+ assert(lazySoi.getAllStructFieldRefs.get(0).getFieldObjectInspector.getTypeName ===
+ "string")
+ assert(lazySoi.getAllStructFieldRefs.get(1).getFieldObjectInspector.getTypeName ===
+ "string")
+ assert(lazySoi.getAllStructFieldRefs.get(2).getFieldObjectInspector.getTypeName ===
+ "array<string>")
+
+ val subclassSchema = hiveIOSchema.copy(
+ outputSerdeClass = Some(classOf[TestLazySimpleSerDe].getCanonicalName))
+ val (_, subclassSoi) = HiveScriptIOSchema.initOutputSerDe(subclassSchema, output).get
+ assert(subclassSoi.getAllStructFieldRefs.get(0).getFieldObjectInspector.getTypeName ===
+ "string")
+
+ SchemaCapturingSerDe.lastColumnTypes = null
+ val customSchema = defaultIOSchema.copy(
+ outputSerdeClass = Some(classOf[SchemaCapturingSerDe].getCanonicalName))
+ HiveScriptIOSchema.initOutputSerDe(customSchema, output)
+ val captured = SchemaCapturingSerDe.lastColumnTypes
+ assert(captured.contains("char(4)"))
+ assert(captured.contains("varchar(5)"))
+ assert(captured.contains("array<char(4)>"))
+ }
+ }
+
+ test("SPARK-59683: preserve-only does not rewrite LazySimpleSerDe CHAR/VARCHAR") {
+ withSQLConf(
+ SQLConf.PRESERVE_CHAR_VARCHAR_TYPE_INFO.key -> "true",
+ SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> "false") {
+ val output = Seq(
+ AttributeReference("c", CharType(4))(),
+ AttributeReference("v", VarcharType(5))(),
+ AttributeReference("nested", ArrayType(CharType(4)))())
+ scala.util.Try(HiveScriptIOSchema.initOutputSerDe(hiveIOSchema, output).get) match {
+ case scala.util.Success((_, soi)) =>
+ val typeName = soi.getAllStructFieldRefs.get(0).getFieldObjectInspector.getTypeName
+ assert(typeName === "char(4)",
+ s"preserve-only must not rewrite CHAR to STRING, found $typeName")
+ assert(soi.getAllStructFieldRefs.get(1).getFieldObjectInspector.getTypeName ===
+ "varchar(5)")
+ assert(soi.getAllStructFieldRefs.get(2).getFieldObjectInspector.getTypeName ===
+ "array<char(4)>")
+ case scala.util.Failure(_) =>
Review Comment:
**Non-blocking (P2):** This catch-all makes any `initOutputSerDe` failure count as success, including unrelated regressions in the schema-selection or initialization code changed by this PR. The test should prove that preserve-only mode supplied the declared CHAR/VARCHAR types, and only tolerate a specifically identified Hive rejection if that outcome is genuinely expected on this lineage.
**Verification:**
- **Compatibility:** Exercise a LazySimpleSerDe subtype under preserve-only mode and assert the exact column-type property supplied to it, with any tolerated failure matched to a specific documented rejection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59683][SQL][HIVE] Support first-class CHAR/VARCHAR in script TRANSFORM output [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor closed pull request #58936: [SPARK-59683][SQL][HIVE] Support first-class CHAR/VARCHAR in script TRANSFORM output
URL: https://github.com/apache/spark/pull/58936
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59683][SQL][HIVE] Support first-class CHAR/VARCHAR in script TRANSFORM output [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #58936:
URL: https://github.com/apache/spark/pull/58936#issuecomment-6031798328
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/6d020ca9295ea5121a831e98ded48dc60061b5f5
- merged into branch-4.x https://github.com/apache/spark/commit/00d66a2e2eb558343e61f4d773d4848a644ea280
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59683][SQL][HIVE] Support first-class CHAR/VARCHAR in script TRANSFORM output [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on code in PR #58936:
URL: https://github.com/apache/spark/pull/58936#discussion_r4199704580
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -62,6 +80,12 @@ trait BaseScriptTransformationExec extends UnaryExecNode {
override def outputPartitioning: Partitioning = child.outputPartitioning
+ // Snapshot at plan construction. CHAR_VARCHAR_STANDARD_SEMANTICS is PERSISTED, so a
+ // plan built under one setting must keep that path even if a later task SQLConf.get
+ // (ReadOnlySQLConf with the executing session) differs. SparkPlan.session is
+ // @transient, so executor `conf` cannot recover the planning session.
+ private val standardCharVarcharSemantics: Boolean = conf.charVarcharStandardSemantics
Review Comment:
Addressed. `ScriptInputOutputSchema` now captures `standardCharVarcharSemantics` at parse under the query or persisted-view SQLConf, and `ScriptTransformationIOSchema` copies that value. The physical operator no longer snapshots `SQLConf` at plan construction.
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationExec.scala:
##########
@@ -268,6 +273,24 @@ object HiveScriptIOSchema extends HiveInspectors {
(columns, columnTypes)
}
+ /**
+ * Hive LazySimpleSerDe CHAR/VARCHAR types truncate on deserialize. Under standard
+ * semantics, map them to STRING so Spark applies length checks. Only LazySimpleSerDe
+ * and subclasses get this rewrite; other SerDes keep the declared schema.
+ * preserveCharVarcharTypeInfo without standard semantics is unchanged.
+ */
+ private def outputTypesForSerDe(
+ serdeClassName: String,
+ columnTypes: Seq[DataType]): Seq[DataType] = {
+ val serdeClass = Utils.classForName[AbstractSerDe](serdeClassName)
+ if (SQLConf.get.charVarcharStandardSemantics &&
Review Comment:
Addressed. Hive LazySimpleSerDe CHAR/VARCHAR rewrite now uses `ioschema.standardCharVarcharSemantics` instead of `SQLConf.get`, so a later session or task conf cannot change the wire schema.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -376,6 +442,106 @@ case class ScriptTransformationIOSchema(
}
object ScriptTransformationIOSchema {
+ private[sql] def toUnboundedStringType(dataType: DataType): DataType = {
+ dataType.transformRecursively {
+ case c: CharType => c.toStringType
+ case v: VarcharType => v.toStringType
+ }
+ }
+
+ // JSON object keys are always strings. Rewrite every map key, including nested maps.
+ // `transformRecursively` would stop at the first matching MapType and skip children.
+ private[sql] def toJsonMapKeyType(dataType: DataType): DataType = dataType match {
+ case ArrayType(et, n) => ArrayType(toJsonMapKeyType(et), n)
+ case MapType(kt, vt, n) =>
+ val jsonKey = if (kt.isInstanceOf[StringType]) kt else StringType
+ MapType(jsonKey, toJsonMapKeyType(vt), n)
+ case StructType(fields) =>
+ StructType(fields.map(f => f.copy(dataType = toJsonMapKeyType(f.dataType))))
+ case other => other
+ }
+
+ /**
+ * Convert a Catalyst value from `sourceType` (JSON or unbounded STRING) to
Review Comment:
Updated the Scaladoc. `sourceType` is the parsed Catalyst type, and maps rebuild through `ArrayBasedMapBuilder` only when the source and target map types differ or the target contains CHAR/VARCHAR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59683][SQL][HIVE] Support first-class CHAR/VARCHAR in script TRANSFORM output [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on PR #58936:
URL: https://github.com/apache/spark/pull/58936#issuecomment-6024079363
Addressed the Oct 5 review in ae74f3ac3ef.
CHAR/VARCHAR mode is now bound on the logical script I/O schema at parse and copied through to execution. No-SerDe conversion and Hive LazySimpleSerDe rewrite both consume that serialized value. Tests cover persisted TRANSFORM views queried under the opposite caller mode, and physical plans executed after flipping the session conf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59683][SQL][HIVE] Support first-class CHAR/VARCHAR in script TRANSFORM output [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58936:
URL: https://github.com/apache/spark/pull/58936#discussion_r4186070318
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -62,6 +80,12 @@ trait BaseScriptTransformationExec extends UnaryExecNode {
override def outputPartitioning: Partitioning = child.outputPartitioning
+ // Snapshot at plan construction. CHAR_VARCHAR_STANDARD_SEMANTICS is PERSISTED, so a
+ // plan built under one setting must keep that path even if a later task SQLConf.get
+ // (ReadOnlySQLConf with the executing session) differs. SparkPlan.session is
+ // @transient, so executor `conf` cannot recover the planning session.
+ private val standardCharVarcharSemantics: Boolean = conf.charVarcharStandardSemantics
Review Comment:
**Non-blocking (P2):** This snapshot is taken too late for the config's persisted-view contract. A view body is resolved under its saved SQLConf, but that scope has ended before the physical script operator is constructed, so a disabled caller makes this field false even when the view was resolved with standard semantics enabled. That can reject no-SerDe scalar output or let the Hive path deserialize through bounded CHAR/VARCHAR types instead of enforcing the saved mode.
See **Shared repair plan 1** in the review body.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:
##########
@@ -376,6 +442,106 @@ case class ScriptTransformationIOSchema(
}
object ScriptTransformationIOSchema {
+ private[sql] def toUnboundedStringType(dataType: DataType): DataType = {
+ dataType.transformRecursively {
+ case c: CharType => c.toStringType
+ case v: VarcharType => v.toStringType
+ }
+ }
+
+ // JSON object keys are always strings. Rewrite every map key, including nested maps.
+ // `transformRecursively` would stop at the first matching MapType and skip children.
+ private[sql] def toJsonMapKeyType(dataType: DataType): DataType = dataType match {
+ case ArrayType(et, n) => ArrayType(toJsonMapKeyType(et), n)
+ case MapType(kt, vt, n) =>
+ val jsonKey = if (kt.isInstanceOf[StringType]) kt else StringType
+ MapType(jsonKey, toJsonMapKeyType(vt), n)
+ case StructType(fields) =>
+ StructType(fields.map(f => f.copy(dataType = toJsonMapKeyType(f.dataType))))
+ case other => other
+ }
+
+ /**
+ * Convert a Catalyst value from `sourceType` (JSON or unbounded STRING) to
Review Comment:
**Nit (P3):** The contract does not match the method below: sourceType is a Catalyst DataType such as ArrayType, MapType, or StructType, while JSON is only the serialized representation. Also, same-typed subtrees without CHAR/VARCHAR return identity before the MapType branch, so unaffected maps are not always rebuilt through ArrayBasedMapBuilder.
**Verification:**
- **Inspection:** Compare the revised source/target terminology and map-rebuild claim with the method signature, early identity return, and MapType branch, confirming every stated guarantee matches the implemented control flow.
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationExec.scala:
##########
@@ -268,6 +273,24 @@ object HiveScriptIOSchema extends HiveInspectors {
(columns, columnTypes)
}
+ /**
+ * Hive LazySimpleSerDe CHAR/VARCHAR types truncate on deserialize. Under standard
+ * semantics, map them to STRING so Spark applies length checks. Only LazySimpleSerDe
+ * and subclasses get this rewrite; other SerDes keep the declared schema.
+ * preserveCharVarcharTypeInfo without standard semantics is unchanged.
+ */
+ private def outputTypesForSerDe(
+ serdeClassName: String,
+ columnTypes: Seq[DataType]): Seq[DataType] = {
+ val serdeClass = Utils.classForName[AbstractSerDe](serdeClassName)
+ if (SQLConf.get.charVarcharStandardSemantics &&
Review Comment:
**Non-blocking (P2):** This re-reads the task's SQLConf even though the script plan has already bound its CHAR/VARCHAR mode. If a plan is materialized with standard semantics enabled and executed after the session disables it, this branch passes bounded CHAR/VARCHAR types to LazySimpleSerDe, so Hive can truncate overlength output before the typed unwrapper can raise EXCEED_LIMIT_LENGTH. The reverse transition can also enable the rewrite for a disabled-mode plan.
See **Shared repair plan 1** in the review body.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org