You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jubin Soni (Jira)" <ji...@apache.org> on 2026/06/17 22:46:00 UTC

[jira] [Updated] (SPARK-57515) Surface MALFORMED_CSV_RECORD instead of ArrayIndexOutOfBoundsException when CSV header exceeds maxColumns

     [ https://issues.apache.org/jira/browse/SPARK-57515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jubin Soni updated SPARK-57515:
-------------------------------
    Priority: Minor  (was: Major)

> Surface MALFORMED_CSV_RECORD instead of ArrayIndexOutOfBoundsException when CSV header exceeds maxColumns
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-57515
>                 URL: https://issues.apache.org/jira/browse/SPARK-57515
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.0.3
>            Reporter: Jubin Soni
>            Priority: Minor
>
> When reading a CSV file with {{header=true}} and the header row contains more columns than {{maxColumns}} (default: {{{}20480{}}}, user-configurable), Spark throws an internal {{java.lang.ArrayIndexOutOfBoundsException}} instead of returning a structured {{MALFORMED_CSV_RECORD}} error.
> This occurs because CSV header validation paths invoke Univocity parsing APIs directly without the malformed-record handling introduced for data rows.
> *Affected Code Paths*
> The issue affects all CSV read paths that validate headers:
>  # *Non-multiLine file read*
>  ** {{CSVHeaderChecker}} calls {{tokenizer.parseLine(header)}} directly.
>  # *MultiLine file read*
>  ** {{CSVHeaderChecker}} calls {{tokenizer.parseNext()}} directly.
>  # *Dataset[String] csv()*
>  ** {{CSVHeaderChecker}} creates a new {{CsvParser}} and calls {{parser.parseLine(line)}} directly.
> In all three cases, a header exceeding {{maxColumns}} surfaces a raw {{{}ArrayIndexOutOfBoundsException{}}}.
> *Background*
> SPARK-57195 (merged 2026-06-14) fixed the same {{ArrayIndexOutOfBoundsException}} issue for CSV data rows by converting parser failures into {{MALFORMED_CSV_RECORD}} errors.
> The SPARK-57195 discussion explicitly noted:
> {quote}Header rows are out of scope from this PR. A header over maxColumns still surfaces the raw AIOOBE (CSVHeaderChecker), a pre-existing gap.
> {quote}
> As a result, header parsing remains inconsistent with data row parsing.
> *Expected Behavior*
> When the header row exceeds {{{}maxColumns{}}}, Spark should fail with a structured {{MALFORMED_CSV_RECORD}} error, consistent with data row handling.
> *Actual Behavior*
> Spark throws:
>  
> {{java.lang.ArrayIndexOutOfBoundsException}}
> originating from Univocity parser internals.
> *Steps to Reproduce*
>  
> {{import java.nio.file.\{Files, Paths}
> import java.nio.charset.StandardCharsets
> val path = "/tmp/test_header.csv"
> Files.write(
>   Paths.get(path),
>   "a,b,c\n1,2,3\n".getBytes(StandardCharsets.UTF_8)
> )
> spark.read
>   .option("header", "true")
>   .option("maxColumns", "2")
>   .csv(path)
>   .collect()}}
> *Result*
>  
> {{java.lang.ArrayIndexOutOfBoundsException}}
> *Expected Result*
> A Spark {{MALFORMED_CSV_RECORD}} error indicating that the CSV record exceeds the configured {{maxColumns}} limit.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org