You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Gang Wu (Jira)" <ji...@apache.org> on 2023/05/11 01:38:00 UTC
[jira] [Commented] (PARQUET-2298) RecordReaderImplementation throw ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/PARQUET-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17721576#comment-17721576 ]
Gang Wu commented on PARQUET-2298:
----------------------------------
The parquet version is kind of old. Could you please try a newer version of Apache Spark which uses higher version of Apache Parquet. If the issue still exists, please also provide the corrupted file if possible.
> RecordReaderImplementation throw ArrayIndexOutOfBoundsException
> ---------------------------------------------------------------
>
> Key: PARQUET-2298
> URL: https://issues.apache.org/jira/browse/PARQUET-2298
> Project: Parquet
> Issue Type: Bug
> Affects Versions: 1.10.1, 1.12.2
> Reporter: Hui An
> Priority: Major
>
> Writing data by spark 3.1(parquet 1.10.1), sometimes it could get the error when reading it.
> {code:java}
> Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 141009 in block 1 in file hdfs://test/test_table/part-01423-58cbf8bc-5b0d-4d53-8279-2c8e3d293cc0.c000.zstd.parquet
> at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:251)
> at org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:207)
> at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:37)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
> at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:93)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
> at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
> at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:177)
> at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
> at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
> at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
> at org.apache.spark.scheduler.Task.run(Task.scala:131)
> at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1509)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 254
> at org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:398)
> at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:226)
> ... 19 more
> {code}
> This table is a wide table(contains 160 columns), and it looks when it try to get a column definition level, which should be 0, but got 254 instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)