You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jorge Molina (Jira)" <ji...@apache.org> on 2026/07/30 23:52:00 UTC
[jira] [Created] (SPARK-58458) No test covers the CSV record split caused by a quoted line break when multiLine is disabled
Jorge Molina created SPARK-58458:
------------------------------------
Summary: No test covers the CSV record split caused by a quoted line break when multiLine is disabled
Key: SPARK-58458
URL: https://issues.apache.org/jira/browse/SPARK-58458
Project: Spark
Issue Type: Test
Components: SQL
Affects Versions: 5.0.0
Reporter: Jorge Molina
A line break inside a quoted value is valid CSV (RFC 4180 §2.6). With the default
`multiLine=false`, the CSV reader cuts such a record at the break: the head keeps its
leading fields with the rest of the schema nulled, and the remainder of the quoted
value starts a new record. The row count rises by one.
`CSVSuite` does not cover this. The suite contains exactly one input literal with a
newline inside quotes, in {{lineSep with 2 chars when multiLine set to ...}}, and it is
confined to the {{multiLine = true}} branch:
{code:scala}
val inputData = if (multiLine) {
s"""name,"i am the${newLine} column1"${newLine}jack,30${newLine}tom,18"""
} else {
s"name,age${newLine}jack,30${newLine}tom,18"
}
{code}
The {{multiLine = false}} branch drops the quoted break, so the default path is never
exercised with the input that distinguishes it. `UnivocityParserSuite` does not cover
it either.
h3. Why it is worth a test
The behaviour follows from the non-multiLine read path being line-oriented by design:
`HadoopFileLinesReader` splits the input on the line separator before the tokenizer
sees it, and `UnivocityParser.parseLine` then receives one physical line at a time.
That is what makes the file splittable, and this ticket does *not* propose changing it.
It proposes pinning what the current design produces, so a future change to that path
cannot alter it silently.
Two consequences are worth locking down in particular:
* {{DROPMALFORMED}} discards *both* halves, so the whole source record is lost rather
than the damaged half — the opposite of what the mode's name suggests for one bad
line.
* {{PERMISSIVE}} without {{columnNameOfCorruptRecord}} — the default — retains both
halves with no signal. The split *adds* a row while rejecting the fragment *removes*
one, so a row-count reconciliation can pass over a record that lost its trailing
fields.
h3. Measured behaviour to be asserted
Measured on `pyspark` 3.5.9 — three records with a quoted line break in the second,
explicit three-column schema. *These are the values to be asserted, not yet confirmed
on `master`* (5.0.0-SNAPSHOT); the fork's CI run is what confirms them, and the PR will
not be opened before it is green.
|| read || rows ||
| default ({{multiLine=false}}) | 4: {{("1","ACME LTDA","SP")}}, {{("2","EMPRESA COM",null)}}, {{("QUEBRA DE LINHA\"","RJ",null)}}, {{("3","OUTRA EMPRESA","MG")}} |
| {{multiLine=true}} | 3, break preserved inside the value |
| {{mode=DROPMALFORMED}} | 2: records 1 and 3; record 2 lost entirely |
| {{columnNameOfCorruptRecord}} declared | 4, both halves carrying their raw text |
PR to follow, test-only, one test added to `CSVSuite`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org