[jira] [Created] (ORC-1060) batch read with Java interface uses high memory when reading ORC string dictionary encoding column

["xiaoli (Jira)" <ji...@apache.org>]

xiaoli created ORC-1060:
---------------------------

             Summary: batch read with Java interface uses high memory when reading ORC string dictionary encoding column
                 Key: ORC-1060
                 URL: https://issues.apache.org/jira/browse/ORC-1060
             Project: ORC
          Issue Type: Improvement
          Components: Java, Reader
    Affects Versions: 1.5.13
            Reporter: xiaoli


We are upgrading spark version from 2.2 to 3.0. During this work, we find spark3.0 uses higher memory than spark2.2 when reading ORC string dictionary encoding column.

The reason is:

spark2.2 use hive's lib to read ORC [https://github.com/aixuebo/hive1.2.1.ql/blob/master/java/org/apache/hadoop/hive/ql/io/orc/TreeReaderFactory.java]  In this code, StringDictionaryTreeReader class with row read interface hold only one string dictionary in memory when reading across multiple file stripes.

spark3.0 use orc lib to read ORC

[https://github.com/apache/orc/blob/main/java/core/src/java/org/apache/orc/impl/TreeReaderFactory.java] In this code, StringDictionaryTreeReader class with batch read interface could hold 3 string dictionary in memory when reading across multiple file stripes:

2 copy of current stripe's dictionary data (dictionaryBuffer variable and dictionaryBufferInBytesCache variable)

and 1 copy of next stripe's dictionary data  (dictionaryBuffer variable, when call

advanceToNextRow method in RecordReaderImpl class's nextBatch method)



--
This message was sent by Atlassian Jira
(v8.20.1#820001)