[jira] [Assigned] (SPARK-60090) TRANSFORM clause and CHAR: V2: Treat like any other STRING source

["Daniel Tenedorio (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-60090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Tenedorio reassigned SPARK-60090:
----------------------------------------

    Assignee: Serge Rielau

> TRANSFORM clause and CHAR: V2: Treat like any other STRING source
> -----------------------------------------------------------------
>
>                 Key: SPARK-60090
>                 URL: https://issues.apache.org/jira/browse/SPARK-60090
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Spark Core
>    Affects Versions: 4.4.0
>            Reporter: Serge Rielau
>            Assignee: Serge Rielau
>            Priority: Major
>              Labels: pull-request-available
>
> Parent / related: SPARK-59683 / PR [#58936|https://github.com/apache/spark/pull/58936] 
> h3. What
> Keep Hive {{TRANSFORM}} / {{MAP}} / {{REDUCE}} script I/O as unbounded {{STRING}} (and JSON for complex types), the same as sources that have no native CHAR/VARCHAR. After the script reader, apply Spark CHAR/VARCHAR assignment ({{{}CharVarcharUtils.stringLengthCheck{}}}: pad {{{}CHAR{}}}, {{EXCEED_LIMIT_LENGTH}} on overflow) in a {{Project}} so the query result type is the declared {{{}CHAR{}}}/{{{}VARCHAR{}}} (including nested, e.g. {{{}MAP<CHAR(5), CHAR(10)>{}}}).
> Do not teach the no-SerDe writers, Hive unwrappers, or SerDe schema rewrite to understand CHAR/VARCHAR.
> h3. Why
> Default Spark (and pre first-class CHAR) already rewrote {{AS (c CHAR(5))}} to {{{}STRING{}}}. No pad, no length check, no {{{}Cast{}}}. Hive TRANSFORM is STRING-on-the-wire. First-class CHAR made {{CharType}} survive parse, so SPARK-59683 built a deep codec inside the script/Hive path.
> That mixed two problems:
>  # CHAR/VARCHAR assignment (pad / length) -- same as CSV/JSON/Parquet.
>  # TRANSFORM JSON / Hive wire issues that are not CHAR-specific (non-string map keys, duplicate JSON keys, SerDe-dependent truncate vs fail). SPARK-59683 only handled some of those on the CHAR path, which is why {{MAP<STRING, STRING>}} vs {{MAP<STRING, CHAR(2)>}} diverged and DATE map keys can round-trip wrong.
> TRANSFORM is Hive-compat, uncommon, documented in OSS Spark, not in Databricks SQL {{{}SELECT{}}}. The CHAR epic already marked Hive TRANSFORM out of scope.
> h3. Proposed plan
>  # Script exec output: recursively replace CHAR/VARCHAR with unbounded STRING (historical {{replaceCharVarcharWithString}} behavior), including nested maps/arrays/structs.
>  # Parse delimited / JSON as today for STRING and complex STRING-shaped types.
>  # Wrap with {{Project}} using {{stringLengthCheck}} (and CHAR pad) onto the declared first-class types when {{spark.sql.charVarchar.standardSemantics.enabled}} is true.
>  # Flag off: keep today's STRING result ({{{}AS (c CHAR(5)){}}} remains {{STRING}} in the schema). No behavior change.
>  # Remove or do not land CHAR-specific logic in {{BaseScriptTransformationExec}} converters, {{outputTypesForSerDe}} LazySimple rewrite, and Hive {{unwrapperFor}} CHAR checks.
>  # JSON non-string map keys ({{{}MAP<INT, ...>{}}}, {{{}MAP<DATE, ...>{}}}): out of scope. Separate ticket: reject unsafe key types, or a reversible codec for all map key types. CHAR map keys are strings on the JSON wire and are handled by assignment.
> h3. Tests / acceptance
>  * Scalar {{AS (c CHAR(n))}} / {{{}VARCHAR(n){}}}: pad, overflow, null token; result type is CHAR/VARCHAR with the flag on; STRING with the flag off.
>  * Nested: {{{}ARRAY<CHAR(n)>{}}}, {{{}STRUCT{}}}, {{MAP<CHAR(n), VARCHAR(m)>}} via the {{{}Project{}}}, not a custom restorer.
>  * Hive SerDe and no-SerDe both go through the same Spark-side assignment (script still STRING-shaped).
>  * Preserve-only ({{{}preserveCharVarcharTypeInfo{}}} without standard semantics): no pad / no {{{}EXCEED_LIMIT_LENGTH{}}}.
>  * No CHAR-only change to map-key dedup vs {{{}MAP<STRING, STRING>{}}}.
> h3. Note for implementer
> SPARK-59683 / OSS PR 58936 should be superseded or stripped back to this model rather than patched further (bind-at-parse flag, Hive unwrapper gating, JSON restorer).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org