[jira] [Created] (HIVE-17243) Replace Hive's DataModel with Java Object Layout

["Prasanth Jayachandran (JIRA)" <ji...@apache.org>]

Prasanth Jayachandran created HIVE-17243:
--------------------------------------------

             Summary: Replace Hive's DataModel with Java Object Layout
                 Key: HIVE-17243
                 URL: https://issues.apache.org/jira/browse/HIVE-17243
             Project: Hive
          Issue Type: Improvement
    Affects Versions: 3.0.0
            Reporter: Prasanth Jayachandran
            Assignee: Prasanth Jayachandran


org.apache.hadoop.hive.ql.util.JavaDataModel is used for estimating memory of objects in many places. The problem with this approach is that manual accounting for all fields and references has to be done. This will also be problematic for cases where shallow object size vs deep object sizes are required. 
Hash table memory monitoring does accounting for size of the hash tables. The estimated sizes of hash tables are often very different for non-vector operators whereas the estimates are close to actual object size for vectorized operators. Also addition of fields requires manual changes to memory monitoring. 

Java object layout is openjdk project that can provide shallow and deep object sizes without using java agents. We can leverage that for much more accurate memory estimates.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)