[jira] [Updated] (HIVE-19824) Improve online datasize estimations for MapJoins

["Zoltan Haindrich (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-19824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zoltan Haindrich updated HIVE-19824:
------------------------------------
    Attachment: HIVE-19824.01wip01.patch

> Improve online datasize estimations for MapJoins
> ------------------------------------------------
>
>                 Key: HIVE-19824
>                 URL: https://issues.apache.org/jira/browse/HIVE-19824
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Zoltan Haindrich
>            Assignee: Zoltan Haindrich
>            Priority: Major
>         Attachments: HIVE-19824.01wip01.patch
>
>
> Statistics.datasize() only accounts for "real" data size; but for example handling 1M rows might introduce some datastructure overhead...if the "real" data is small - even this overhead might become the real memory usage
> for 6.5M rows of (int,int) the estimation is 52MB
> in reality this eats up ~260MB from which 210MB is used to service the hashmap functionality to that many rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)