[jira] [Resolved] (SPARK-60102) Apply CSV CHAR/VARCHAR assignment after STRING parse

["Serge Rielau (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-60102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Serge Rielau resolved SPARK-60102.
----------------------------------
    Resolution: Not A Problem

Thsi PR is not needed.

>  Apply CSV CHAR/VARCHAR assignment after STRING parse
> -----------------------------------------------------
>
>                 Key: SPARK-60102
>                 URL: https://issues.apache.org/jira/browse/SPARK-60102
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Spark Core
>    Affects Versions: 5.0.0
>            Reporter: Serge Rielau
>            Priority: Major
>              Labels: pull-request-available
>
> Context
>  
> SPARK-59274 applies CHAR/VARCHAR assignment inside UnivocityParser via applyTextParseSemantics on each STRING field. SPARK-59722 (JSON) and SPARK-60090 (Hive TRANSFORM) instead parse as unbounded STRING, then wrap the result with CharVarcharUtils.assignAfterParse / stringLengthCheck.
>  
> CSV is the next format. It has no map keys and little CHAR-specific recovery, so it is the check that assignAfterParse is not JSON-shaped.
>  
> Do not wait on SPARK-60090 if those helpers are already on master via SPARK-59722 (replaceCharVarcharWithStringAlways, replaceCharVarcharWithStringInSchemaAlways, assignAfterParse). Do not combine this with XML.
>  
> Proposed change
>  
> For from_csv and the CSV datasource:
>  
> * Build UnivocityParser converters from a physical schema in which every CHAR/VARCHAR is replaced by STRING (replaceCharVarcharWithStringAlways), including when first-class CHAR/VARCHAR types are enabled.
> * After a row is parsed, apply CharVarcharUtils.assignAfterParse (write-side pad / EXCEED_LIMIT_LENGTH).
> * Remove applyTextParseSemantics from the CSV STRING converter.
> * Skip CSV filter pushdown when the schema contains CHAR/VARCHAR, matching JacksonParser after SPARK-59722.
>  
> Keep the wrapping reusable (not a JSON-only InternalRow shape). CSV has no MAP keys, so MapFromArrays / mapKeyDedupPolicy is out of scope.
>  
> User-facing change
>  
> Only when spark.sql.charVarchar.standardSemantics.enabled is true (still false by default):
>  
> * CHAR padding and VARCHAR/CHAR overflow still apply to from_csv and CSV reads.
> * Overflow becomes a parse-mode error for the whole parsed row (PERMISSIVE: null; FAILFAST: EXCEED_LIMIT_LENGTH), matching JSON after SPARK-59722. SPARK-59274 currently fails the overflowing field only.
>  
> Document in the 4.3-to-4.4 SQL migration guide.
> Tests
>  
> Extend BasicCharVarcharTestSuite for from_csv + CSV datasource: padding, overflow under PERMISSIVE/FAILFAST, nested STRUCT/ARRAY, no change with the flag off. Keep SPARK-59274 CSV document-as-CHAR cases (padding vs delimiter).
>  
> Related
>  
> * SPARK-59274 (in-parser CHAR/VARCHAR for JSON/CSV/XML)
> * SPARK-59722 (JSON post-parse assignment)
> * SPARK-60090 (Hive TRANSFORM post-parse assignment)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org