You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Chao Sun (Jira)" <ji...@apache.org> on 2026/03/11 23:37:00 UTC
[jira] [Created] (SPARK-55968) Spark should not treat all RuntimeException as corrupted file when `spark.sql.files.ignoreCorruptFiles` is enabled
Chao Sun created SPARK-55968:
--------------------------------
Summary: Spark should not treat all RuntimeException as corrupted file when `spark.sql.files.ignoreCorruptFiles` is enabled
Key: SPARK-55968
URL: https://issues.apache.org/jira/browse/SPARK-55968
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1, 3.5.8, 4.0.2
Reporter: Chao Sun
When {{spark.sql.files.ignoreCorruptFiles}} is enabled, Spark currently will catch any {{RuntimeException}} and ignore the data files. See [FileScanRDD|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala#L271-L276] and [DataSourceUtils|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala#L202-L205]
In particular, sometimes vectorized Parquet reader could fail with the following error
{code:java}
java.lang.RuntimeException: Cannot reserve additional contiguous bytes in the vectorized reader (integer overflow). As a workaround, you can reduce the vectorized reader batch size, or disable the vectorized reader, or disable spark.sql.sources.bucketing.enabled if you read from bucket table. For Parquet file format, refer to spark.sql.parquet.columnarReaderBatchSize (default 4096) and spark.sql.parquet.enableVectorizedReader; for ORC file format, refer to spark.sql.orc.columnarReaderBatchSize (default 4096) and spark.sql.orc.enableVectorizedReader.
{code}
which would still be treated as data corruption. This would silently cause data loss.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org