[jira] [Created] (HIVE-10161) LLAP: IO buffers seem to be hard-coded to 256kb

["Gopal V (JIRA)" <ji...@apache.org>]

Gopal V created HIVE-10161:
------------------------------

             Summary: LLAP: IO buffers seem to be hard-coded to 256kb 
                 Key: HIVE-10161
                 URL: https://issues.apache.org/jira/browse/HIVE-10161
             Project: Hive
          Issue Type: Sub-task
    Affects Versions: llap
            Reporter: Gopal V
            Assignee: Sergey Shelukhin
             Fix For: llap


The EncodedReaderImpl will die when reading from the cache, when reading data written by the regular ORC writer 

{code}
Caused by: java.io.IOException: java.lang.IllegalArgumentException: Buffer size too small. size = 262144 needed = 3919246
        at org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:249)
        at org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:201)
        at org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:140)
        at org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:96)
        at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
        ... 22 more
Caused by: java.lang.IllegalArgumentException: Buffer size too small. size = 262144 needed = 3919246
        at org.apache.hadoop.hive.ql.io.orc.InStream.addOneCompressionBuffer(InStream.java:780)
        at org.apache.hadoop.hive.ql.io.orc.InStream.uncompressStream(InStream.java:628)
        at org.apache.hadoop.hive.ql.io.orc.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:309)
        at org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:278)
        at org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:48)
        at org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
        ... 4 more
]], Vertex failed as one or more tasks failed. failedTasks:1, Vertex vertex_1424502260528_1945_1_00 [Map 1] killed/failed due to:null]
{code}

Turning off hive.llap.io.enabled makes the error go away.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)