You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "Yukihiro Okada (Jira)" <ji...@apache.org> on 2019/10/10 12:43:00 UTC
[jira] [Comment Edited] (ORC-557) Large ORC file parsing failed
[ https://issues.apache.org/jira/browse/ORC-557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948550#comment-16948550 ]
Yukihiro Okada edited comment on ORC-557 at 10/10/19 12:42 PM:
---------------------------------------------------------------
Reproduced this bug with master branch.
{code:shell}
❯ ls -lh /tmp/output.000.orc
-rw-r--r-- 1 yuokada wheel 4.5G 10 10 21:31 /tmp/output.000.orc
❯ java -jar tools/target/orc-tools-1.7.0-SNAPSHOT-uber.jar meta /tmp/output.000.orc
Processing data file /tmp/output.000.orc [length: 4805398749]
Exception in thread "main" java.io.IOException: Problem reading file footer /tmp/output.000.orc
at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:717)
at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:500)
at org.apache.orc.OrcFile.createReader(OrcFile.java:365)
at org.apache.orc.tools.FileDump.getReader(FileDump.java:241)
at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:333)
at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:274)
at org.apache.orc.tools.FileDump.main(FileDump.java:135)
at org.apache.orc.tools.Driver.main(Driver.java:108)
Caused by: java.lang.IllegalArgumentException: newPosition > limit: (43291 > 29438)
at java.base/java.nio.Buffer.createPositionException(Buffer.java:318)
at java.base/java.nio.Buffer.position(Buffer.java:293)
at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1086)
at org.apache.orc.impl.InStream$UncompressedStream.setCurrent(InStream.java:134)
at org.apache.orc.impl.InStream$UncompressedStream.reset(InStream.java:110)
at org.apache.orc.impl.InStream$UncompressedStream.<init>(InStream.java:100)
at org.apache.orc.impl.InStream.create(InStream.java:844)
at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:706)
... 7 more
{code}
was (Author: yuokada):
Reproduced this bug.
{code:shell}
❯ ls -lh /tmp/output.000.orc
-rw-r--r-- 1 yuokada wheel 4.5G 10 10 21:31 /tmp/output.000.orc
❯ java -jar tools/target/orc-tools-1.7.0-SNAPSHOT-uber.jar meta /tmp/output.000.orc
Processing data file /tmp/output.000.orc [length: 4805398749]
Exception in thread "main" java.io.IOException: Problem reading file footer /tmp/output.000.orc
at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:717)
at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:500)
at org.apache.orc.OrcFile.createReader(OrcFile.java:365)
at org.apache.orc.tools.FileDump.getReader(FileDump.java:241)
at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:333)
at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:274)
at org.apache.orc.tools.FileDump.main(FileDump.java:135)
at org.apache.orc.tools.Driver.main(Driver.java:108)
Caused by: java.lang.IllegalArgumentException: newPosition > limit: (43291 > 29438)
at java.base/java.nio.Buffer.createPositionException(Buffer.java:318)
at java.base/java.nio.Buffer.position(Buffer.java:293)
at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1086)
at org.apache.orc.impl.InStream$UncompressedStream.setCurrent(InStream.java:134)
at org.apache.orc.impl.InStream$UncompressedStream.reset(InStream.java:110)
at org.apache.orc.impl.InStream$UncompressedStream.<init>(InStream.java:100)
at org.apache.orc.impl.InStream.create(InStream.java:844)
at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:706)
... 7 more
{code}
> Large ORC file parsing failed
> -----------------------------
>
> Key: ORC-557
> URL: https://issues.apache.org/jira/browse/ORC-557
> Project: ORC
> Issue Type: Bug
> Components: Reader, tools
> Affects Versions: 1.6.0, 1.6.1
> Reporter: 周娜
> Priority: Major
> Attachments: image-2019-10-09-15-36-48-079.png
>
>
> When writing a more than 4G ORC file. The following error will occur when dump the file：
> !image-2019-10-09-15-36-48-079.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)