[jira] [Updated] (HIVE-14130) HCatalog improvement by reducing invocations of toLowerCase() for fieldNames and repeatedly using DefaultHCatRecord in HCatRecordReader, and adding static fields in HCatRecordSerDe.java

["Zhu Li (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-14130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zhu Li updated HIVE-14130:
--------------------------
    Summary: HCatalog improvement by reducing invocations of toLowerCase() for fieldNames and repeatedly using DefaultHCatRecord in HCatRecordReader, and adding static fields in  HCatRecordSerDe.java  (was: HCatalog improvement by reducing invocations of toLowerCase() for fieldNames, repeatedly using DefaultHCatRecord, and adding static fields in  HCatRecordSerDe.java)

> HCatalog improvement by reducing invocations of toLowerCase() for fieldNames and repeatedly using DefaultHCatRecord in HCatRecordReader, and adding static fields in  HCatRecordSerDe.java
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-14130
>                 URL: https://issues.apache.org/jira/browse/HIVE-14130
>             Project: Hive
>          Issue Type: Improvement
>          Components: HCatalog
>            Reporter: Zhu Li
>            Assignee: Zhu Li
>              Labels: patch, performance
>   Original Estimate: 216h
>  Remaining Estimate: 216h
>
> 1. In HCatalog,  the code used for lazy deserialization in HCatRecordReader.java uses a method named getPosition(fieldName) for getting index of a filed in a row. When it is invoked, it also invokes toLowerCase() method for the String variable fieldName. This is trivial when data size is small, but when data size is huge, repeated invocations of toLowerCase() for the same set of fieldNames wastes some time. So storing the indices for the columns names in HcatRecordReader class or storing lower-case fieldNames in outputSchema will improve efficiency. 
> 2. HCatRecordReader.java is creating new instance of DefaultHCatRecord repeatedly for every new incoming row of data. This causes a waste of time. Adding a private variable of DefaultHCatRecord in this class and using it repeatedly for new rows will reduce some overhead.
> 3. Method serializePrimitiveField in class HCatRecordSerDe.java is invoking 
> HCatContext.INSTANCE.getConf() repeatedly. This also causes some overhead according to result by JProfiler. Adding a static boolean field in HCatRecordSerDe.java which stores HCatContext.INSTANCE.getConf().isPresent() and another static Configuration variable which stores result of HCatContext.INSTANCE.getConf() also reduces overhead.
>  According to my test on a cluster, using the above modifications we can save 80 seconds or so when HCatalog is used to load a table in size of 1 billion(rows) * 40(columns) with various data types. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)