You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2026/06/14 17:59:00 UTC
[jira] [Resolved] (SPARK-57195) CSV multiLine schema inference throws ArrayIndexOutOfBoundsException for a row exceeding maxColumns
[ https://issues.apache.org/jira/browse/SPARK-57195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-57195.
---------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 56260
[https://github.com/apache/spark/pull/56260]
> CSV multiLine schema inference throws ArrayIndexOutOfBoundsException for a row exceeding maxColumns
> ---------------------------------------------------------------------------------------------------
>
> Key: SPARK-57195
> URL: https://issues.apache.org/jira/browse/SPARK-57195
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.2, 3.5.8, 4.1.2
> Reporter: Yash Botadra
> Assignee: Yash Botadra
> Priority: Major
> Labels: correctness, pull-request-available
> Fix For: 4.3.0
>
>
> When inferring the schema of a CSV with multiLine=true and inferSchema=true, a data row with more columns than maxColumns causes a raw java.lang.ArrayIndexOutOfBoundsException to propagate and fail the query, instead of a user-facing error.
> SPARK-49444 made the per-line parse path (UnivocityParser.parseLine) translate this into a MALFORMED_CSV_RECORD error. The streaming path used by multiLine reads and schema inference (UnivocityParser.tokenizeStream / convertStream, via tokenizer.parseNext()) was not covered, so the same input still throws a raw ArrayIndexOutOfBoundsException.
>
> Repro:
> {code:java}
> file:
> a,b
> c,d
> 1,2,3
> spark.read.option("header","false").option("inferSchema","true")
> .option("multiLine","true").option("maxColumns","2").csv(path)
> Expected: MALFORMED_CSV_RECORD (SQLSTATE KD000).
> Actual: java.lang.ArrayIndexOutOfBoundsException.
> {code}
>
> How introduced: the streaming parseNext() path predates SPARK-49444 and was missed when that fix was applied to parseLine.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org