You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2018/02/13 18:07:00 UTC
[jira] [Commented] (ORC-285) Empty vector batches of floats or
doubles get java.io.EOFException
[ https://issues.apache.org/jira/browse/ORC-285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362759#comment-16362759 ]
Owen O'Malley commented on ORC-285:
-----------------------------------
For the Spark project, I looked into the background of this bug. My findings are:
After a bit of digging, the problem that ORC-285 fixes was introduced in HIVE-9711. HIVE-9711 is included in Hive 1.2.0, but doesn't create a problem unless the vectorized ORC reader is used. That didn't become the default behavior until HIVE-11417, which was released in Hive 2.1.0.
> Empty vector batches of floats or doubles get java.io.EOFException
> -------------------------------------------------------------------
>
> Key: ORC-285
> URL: https://issues.apache.org/jira/browse/ORC-285
> Project: ORC
> Issue Type: Bug
> Affects Versions: 1.3.4, 1.4.1
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Priority: Major
> Fix For: 1.4.3, 1.5.0
>
>
> The FloatTreeReader and DoubleTreeReader both fail if given a set of empty values. This often happens when the float or double data is inside a list where all of the values are empty.
> The stack trace looks like:
> {code}
> java.io.IOException: Error reading file: /Users/piyush.mukati/tmp/11-task_1511779500016_298243_r_000031-r-00031.orc
> at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1191)
> at org.apache.orc.tools.ScanData.main(ScanData.java:67)
> at org.apache.orc.tools.Driver.main(Driver.java:109)
> Caused by: java.io.EOFException: Read past EOF for compressed stream Stream for column 36 kind DATA position: 0 length: 0 range: 0 offset: 0 limit: 0
> at org.apache.orc.impl.SerializationUtils.readFully(SerializationUtils.java:119)
> at org.apache.orc.impl.SerializationUtils.readLongLE(SerializationUtils.java:102)
> at org.apache.orc.impl.SerializationUtils.readDouble(SerializationUtils.java:98)
> at org.apache.orc.impl.TreeReaderFactory$DoubleTreeReader.nextVector(TreeReaderFactory.java:763)
> at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.nextVector(TreeReaderFactory.java:1835)
> at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.nextVector(TreeReaderFactory.java:1835)
> at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.nextVector(TreeReaderFactory.java:1835)
> at org.apache.orc.impl.TreeReaderFactory$ListTreeReader.nextVector(TreeReaderFactory.java:2003)
> at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.nextVector(TreeReaderFactory.java:1835)
> at org.apache.orc.impl.TreeReaderFactory$ListTreeReader.nextVector(TreeReaderFactory.java:2003)
> at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.nextVector(TreeReaderFactory.java:1835)
> at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.nextBatch(TreeReaderFactory.java:1817)
> at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1184)
> ... 2 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)