You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jubin Soni (Jira)" <ji...@apache.org> on 2026/06/17 22:46:00 UTC
[jira] [Updated] (SPARK-57515) Surface MALFORMED_CSV_RECORD instead of ArrayIndexOutOfBoundsException when CSV header exceeds maxColumns
[ https://issues.apache.org/jira/browse/SPARK-57515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jubin Soni updated SPARK-57515:
-------------------------------
Priority: Minor (was: Major)
> Surface MALFORMED_CSV_RECORD instead of ArrayIndexOutOfBoundsException when CSV header exceeds maxColumns
> ---------------------------------------------------------------------------------------------------------
>
> Key: SPARK-57515
> URL: https://issues.apache.org/jira/browse/SPARK-57515
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.3
> Reporter: Jubin Soni
> Priority: Minor
>
> When reading a CSV file with {{header=true}} and the header row contains more columns than {{maxColumns}} (default: {{{}20480{}}}, user-configurable), Spark throws an internal {{java.lang.ArrayIndexOutOfBoundsException}} instead of returning a structured {{MALFORMED_CSV_RECORD}} error.
> This occurs because CSV header validation paths invoke Univocity parsing APIs directly without the malformed-record handling introduced for data rows.
> *Affected Code Paths*
> The issue affects all CSV read paths that validate headers:
> # *Non-multiLine file read*
> ** {{CSVHeaderChecker}} calls {{tokenizer.parseLine(header)}} directly.
> # *MultiLine file read*
> ** {{CSVHeaderChecker}} calls {{tokenizer.parseNext()}} directly.
> # *Dataset[String] csv()*
> ** {{CSVHeaderChecker}} creates a new {{CsvParser}} and calls {{parser.parseLine(line)}} directly.
> In all three cases, a header exceeding {{maxColumns}} surfaces a raw {{{}ArrayIndexOutOfBoundsException{}}}.
> *Background*
> SPARK-57195 (merged 2026-06-14) fixed the same {{ArrayIndexOutOfBoundsException}} issue for CSV data rows by converting parser failures into {{MALFORMED_CSV_RECORD}} errors.
> The SPARK-57195 discussion explicitly noted:
> {quote}Header rows are out of scope from this PR. A header over maxColumns still surfaces the raw AIOOBE (CSVHeaderChecker), a pre-existing gap.
> {quote}
> As a result, header parsing remains inconsistent with data row parsing.
> *Expected Behavior*
> When the header row exceeds {{{}maxColumns{}}}, Spark should fail with a structured {{MALFORMED_CSV_RECORD}} error, consistent with data row handling.
> *Actual Behavior*
> Spark throws:
>
> {{java.lang.ArrayIndexOutOfBoundsException}}
> originating from Univocity parser internals.
> *Steps to Reproduce*
>
> {{import java.nio.file.\{Files, Paths}
> import java.nio.charset.StandardCharsets
> val path = "/tmp/test_header.csv"
> Files.write(
> Paths.get(path),
> "a,b,c\n1,2,3\n".getBytes(StandardCharsets.UTF_8)
> )
> spark.read
> .option("header", "true")
> .option("maxColumns", "2")
> .csv(path)
> .collect()}}
> *Result*
>
> {{java.lang.ArrayIndexOutOfBoundsException}}
> *Expected Result*
> A Spark {{MALFORMED_CSV_RECORD}} error indicating that the CSV record exceeds the configured {{maxColumns}} limit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org