You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Gopal V (JIRA)" <ji...@apache.org> on 2018/08/16 07:53:00 UTC
[jira] [Comment Edited] (HIVE-20321) Vectorization: Cut down memory
size of 1 col VectorHashKeyWrapper to <1 CacheLine
[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582110#comment-16582110 ]
Gopal V edited comment on HIVE-20321 at 8/16/18 7:52 AM:
---------------------------------------------------------
LGTM - +1
The single key ::equals() implementation has been improved by 15% across the whole query (& dropped out of the top-5 functions).
The only test query which did not improve was {{select count(distinct ss_item_sk), count(distinct ss_customer_sk) from store_sales;}}
was (Author: gopalv):
LGTM - +1
The single key ::equals() implementation has been improved by 15% across the whole query I tested with (& dropped out of the top-5 functions).
The only test query which did not improve was {{select count(distinct ss_item_sk), count(distinct ss_customer_sk) from store_sales;}}
> Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
> ---------------------------------------------------------------------------------
>
> Key: HIVE-20321
> URL: https://issues.apache.org/jira/browse/HIVE-20321
> Project: Hive
> Issue Type: Bug
> Reporter: Gopal V
> Assignee: Matt McCline
> Priority: Major
> Attachments: HIVE-20321.01.patch, HIVE-20321.02.patch, HIVE-20321.03.patch
>
>
> With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper is bigger than the low Xmx JVMs.
> {code}
> ***** 64-bit VM: **********************************************************
> org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals:
> OFFSET SIZE TYPE DESCRIPTION VALUE
> 0 16 (object header) N/A
> 16 4 int VectorHashKeyWrapper.hashcode N/A
> 20 4 (alignment/padding gap)
> 24 8 long[] VectorHashKeyWrapper.longValues N/A
> 32 8 double[] VectorHashKeyWrapper.doubleValues N/A
> 40 8 byte[][] VectorHashKeyWrapper.byteValues N/A
> 48 8 int[] VectorHashKeyWrapper.byteStarts N/A
> 56 8 int[] VectorHashKeyWrapper.byteLengths N/A
> 64 8 org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[] VectorHashKeyWrapper.decimalValues N/A
> 72 8 java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A
> 80 8 org.apache.hadoop.hive.common.type.HiveIntervalDayTime[] VectorHashKeyWrapper.intervalDayTimeValues N/A
> 88 8 boolean[] VectorHashKeyWrapper.isNull N/A
> 96 8 org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext VectorHashKeyWrapper.hashCtx N/A
> Instance size: 104 bytes
> Space losses: 4 bytes internal + 0 bytes external = 4 bytes total
> {code}
> Pulling this up to a parent class allows for this to be cut down to 32 bytes for the single column case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)