You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "zhaolong (Jira)" <ji...@apache.org> on 2025/05/09 10:22:00 UTC
[jira] [Updated] (ORC-1897) ORC file Damaged has many different exception
[ https://issues.apache.org/jira/browse/ORC-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhaolong updated ORC-1897:
--------------------------
Description:
We have find many cases of ORC file corruption, and errors will be reported when reading.
# java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.orc.impl.RunLengthIntegerReaderV2.readPatchedBaseValues(RunLengthIntegerReaderV2.java:200) at org.apache.orc.impl.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:70) at org.apache.orc.impl.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:323) at org.apache.orc.impl.RunLengthIntegerReaderV2.nextVector(RunLengthIntegerReaderV2.java:373) at org.apache.orc.impl.TreeReaderFactory$LongTreeReader.nextVector(TreeReaderFactory.java:696) at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.nextVector(TreeReaderFactory.java:2463) at org.apache.orc.impl.reader.tree.StructBatchReader.readBatchColumn(StructBatchReader.java:42) at org.apache.orc.impl.reader.tree.StructBatchReader.nextBatch(StructBatchReader.java:72) at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1236)
# java.lang.IllegalArgumentException: Buffer size too small. size 131072 needed = 471700 in column 1 kind DICTIONARY DATA
at org.apache.orc.impl,InStream$CompressedStream.readHeader(InStream.java:487)
at org apache.orc.impl.InStream$CompressedStream.ensureUncompressed(InStream.java: 531)
at org.apache.orc.impl:InStream$CompressedStream.available(InStream.java:538)
at org apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.readDictionaryStream(TreeReaderFactory.java:1776)
at org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.startStripe(TreeReaderFactory. java:1740)
at org.apache.orc.impl.TreeReaderFactory$StringTreeReader.startStripe(TreeReaderFactory. java: 1491)
at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.startStripe(TreeReaderFactory. java:2076)
at org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl. java: 1117)
at org apache.orc impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java: 1154)
at org apache.orc impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1189)
at org.apache.orc.impl,RecordReaderImpl,<init>(RecordReaderImpl, java:251)
at org apache.orc impl.ReaderImpl.rows(ReaderImpl,java:851)
at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:845)
Take the second figure as an example. The chunkLength should be between 32k (snappy) and 256k (lzo, zlib), but why needed = 471700. We have tested the CPU and memory of the hardware, and no error is found. The EC policy of the HDFS is also not configured.
So we want to read the orc file after hive write orc file in filesinkoperator. However, considering the performance impact, we can only read orc metadata such as stripe size to check whether there is any problem.
Is there any other way to solve the above problem?
was:
We have find many cases of ORC file corruption, and errors will be reported when reading.
# java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.orc.impl.RunLengthIntegerReaderV2.readPatchedBaseValues(RunLengthIntegerReaderV2.java:200) at org.apache.orc.impl.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:70) at org.apache.orc.impl.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:323) at org.apache.orc.impl.RunLengthIntegerReaderV2.nextVector(RunLengthIntegerReaderV2.java:373) at org.apache.orc.impl.TreeReaderFactory$LongTreeReader.nextVector(TreeReaderFactory.java:696) at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.nextVector(TreeReaderFactory.java:2463) at org.apache.orc.impl.reader.tree.StructBatchReader.readBatchColumn(StructBatchReader.java:42) at org.apache.orc.impl.reader.tree.StructBatchReader.nextBatch(StructBatchReader.java:72) at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1236)
# java.lang.IllegalArgumentException: Buffer size too small. size 131072 needed = 471700 in column 1 kind DICTIONARY DATA
at org.apache.orc.impl,InStream$CompressedStream.readHeader(InStream.java:487)
at org apache.orc.impl.InStream$CompressedStream.ensureUncompressed(InStream.java: 531)
at org.apache.orc.impl:InStream$CompressedStream.available(InStream.java:538)
at org apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.readDictionaryStream(TreeReaderFactory.java:1776)
at org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.startStripe(TreeReaderFactory. java:1740)
at org.apache.orc.impl.TreeReaderFactory$StringTreeReader.startStripe(TreeReaderFactory. java: 1491)
at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.startStripe(TreeReaderFactory. java:2076)
at org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl. java: 1117)
at org apache.orc impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java: 1154)
at org apache.orc impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1189)
at org.apache.orc.impl,RecordReaderImpl,<init>(RecordReaderImpl, java:251)
at org apache.orc impl.ReaderImpl.rows(ReaderImpl,java:851)
at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:845)
Take the second figure as an example. The chunkLength should be between 32k (snappy) and 256k (lzo, zlib), but why needed = 471700. We have tested the CPU and memory of the hardware, and no error is found. The EC policy of the HDFS is not configured.
So we want to read the orc file after hive write orc file in filesinkoperator. However, considering the performance impact, we can only read orc metadata such as stripe size to check whether there is any problem.
> ORC file Damaged has many different exception
> ---------------------------------------------
>
> Key: ORC-1897
> URL: https://issues.apache.org/jira/browse/ORC-1897
> Project: ORC
> Issue Type: Bug
> Affects Versions: 1.6.7, 2.1.2
> Reporter: zhaolong
> Priority: Blocker
>
> We have find many cases of ORC file corruption, and errors will be reported when reading.
> # java.lang.ArrayIndexOutOfBoundsException: 0 at org.apache.orc.impl.RunLengthIntegerReaderV2.readPatchedBaseValues(RunLengthIntegerReaderV2.java:200) at org.apache.orc.impl.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:70) at org.apache.orc.impl.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:323) at org.apache.orc.impl.RunLengthIntegerReaderV2.nextVector(RunLengthIntegerReaderV2.java:373) at org.apache.orc.impl.TreeReaderFactory$LongTreeReader.nextVector(TreeReaderFactory.java:696) at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.nextVector(TreeReaderFactory.java:2463) at org.apache.orc.impl.reader.tree.StructBatchReader.readBatchColumn(StructBatchReader.java:42) at org.apache.orc.impl.reader.tree.StructBatchReader.nextBatch(StructBatchReader.java:72) at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1236)
> # java.lang.IllegalArgumentException: Buffer size too small. size 131072 needed = 471700 in column 1 kind DICTIONARY DATA
> at org.apache.orc.impl,InStream$CompressedStream.readHeader(InStream.java:487)
> at org apache.orc.impl.InStream$CompressedStream.ensureUncompressed(InStream.java: 531)
> at org.apache.orc.impl:InStream$CompressedStream.available(InStream.java:538)
> at org apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.readDictionaryStream(TreeReaderFactory.java:1776)
> at org.apache.orc.impl.TreeReaderFactory$StringDictionaryTreeReader.startStripe(TreeReaderFactory. java:1740)
> at org.apache.orc.impl.TreeReaderFactory$StringTreeReader.startStripe(TreeReaderFactory. java: 1491)
> at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.startStripe(TreeReaderFactory. java:2076)
> at org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl. java: 1117)
> at org apache.orc impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java: 1154)
> at org apache.orc impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1189)
> at org.apache.orc.impl,RecordReaderImpl,<init>(RecordReaderImpl, java:251)
> at org apache.orc impl.ReaderImpl.rows(ReaderImpl,java:851)
> at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:845)
> Take the second figure as an example. The chunkLength should be between 32k (snappy) and 256k (lzo, zlib), but why needed = 471700. We have tested the CPU and memory of the hardware, and no error is found. The EC policy of the HDFS is also not configured.
> So we want to read the orc file after hive write orc file in filesinkoperator. However, considering the performance impact, we can only read orc metadata such as stripe size to check whether there is any problem.
> Is there any other way to solve the above problem?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)