[jira] [Updated] (SPARK-58458) No test covers the CSV record split caused by a quoted line break when multiLine is disabled

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-58458:
-----------------------------------
    Labels: pull-request-available  (was: )

> No test covers the CSV record split caused by a quoted line break when multiLine is disabled
> --------------------------------------------------------------------------------------------
>
>                 Key: SPARK-58458
>                 URL: https://issues.apache.org/jira/browse/SPARK-58458
>             Project: Spark
>          Issue Type: Test
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Jorge Molina
>            Priority: Major
>              Labels: pull-request-available
>
> A line break inside a quoted value is valid CSV (RFC 4180 §2.6). With the default
> `multiLine=false`, the CSV reader cuts such a record at the break: the head keeps its
> leading fields with the rest of the schema nulled, and the remainder of the quoted
> value starts a new record. The row count rises by one.
> `CSVSuite` does not cover this. The suite contains exactly one input literal with a
> newline inside quotes, in {{lineSep with 2 chars when multiLine set to ...}}, and it is
> confined to the {{multiLine = true}} branch:
> {code:scala}
> val inputData = if (multiLine) {
>   s"""name,"i am the${newLine} column1"${newLine}jack,30${newLine}tom,18"""
> } else {
>   s"name,age${newLine}jack,30${newLine}tom,18"
> }
> {code}
> The {{multiLine = false}} branch drops the quoted break, so the default path is never
> exercised with the input that distinguishes it. `UnivocityParserSuite` does not cover
> it either.
> h3. Why it is worth a test
> The behaviour follows from the non-multiLine read path being line-oriented by design:
> `HadoopFileLinesReader` splits the input on the line separator before the tokenizer
> sees it, and `UnivocityParser.parseLine` then receives one physical line at a time.
> That is what makes the file splittable, and this ticket does *not* propose changing it.
> It proposes pinning what the current design produces, so a future change to that path
> cannot alter it silently.
> Two consequences are worth locking down in particular:
> * {{DROPMALFORMED}} discards *both* halves, so the whole source record is lost rather
>   than the damaged half — the opposite of what the mode's name suggests for one bad
>   line.
> * {{PERMISSIVE}} without {{columnNameOfCorruptRecord}} — the default — retains both
>   halves with no signal. The split *adds* a row while rejecting the fragment *removes*
>   one, so a row-count reconciliation can pass over a record that lost its trailing
>   fields.
> h3. Measured behaviour to be asserted
> Measured on `pyspark` 3.5.9 — three records with a quoted line break in the second,
> explicit three-column schema. *These are the values to be asserted, not yet confirmed
> on `master`* (5.0.0-SNAPSHOT); the fork's CI run is what confirms them, and the PR will
> not be opened before it is green.
> || read || rows ||
> | default ({{multiLine=false}}) | 4: {{("1","ACME LTDA","SP")}}, {{("2","EMPRESA COM",null)}}, {{("QUEBRA DE LINHA\"","RJ",null)}}, {{("3","OUTRA EMPRESA","MG")}} |
> | {{multiLine=true}} | 3, break preserved inside the value |
> | {{mode=DROPMALFORMED}} | 2: records 1 and 3; record 2 lost entirely |
> | {{columnNameOfCorruptRecord}} declared | 4, both halves carrying their raw text |
> PR to follow, test-only, one test added to `CSVSuite`.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org