You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Zoltan Haindrich (JIRA)" <ji...@apache.org> on 2018/12/04 09:03:00 UTC
[jira] [Comment Edited] (HIVE-21000) Upgrade thrift to at least
0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708419#comment-16708419 ]
Zoltan Haindrich edited comment on HIVE-21000 at 12/4/18 9:02 AM:
------------------------------------------------------------------
we might also improve on this by also caching the hashcode somewhere...but as long as these objects are mutable; it's hard to not introduce bugs by doing that....
Replacing only the hashcode implementation of {{FieldSchema}} with a non-list based one reduced my actual query compile time from 3.3sec to 1.5sec ; time spent in FieldSchema.hashCode - have gone down from 22secs to 3sec (query was compile 10 times or so)
So I think this is really worth the effort
was (Author: kgyrtkirk):
we might also improve on this by also caching the hashcode somewhere...but as long as these objects are mutable; it's hard to not introduce bugs by doing that....
Replacing only the hashcode implementation of {{FieldSchema}} with a non-list based one reduced my actual query compile time from 3.3sec to 1.5sec - so I think this is really worth it.
> Upgrade thrift to at least 0.10.0
> ---------------------------------
>
> Key: HIVE-21000
> URL: https://issues.apache.org/jira/browse/HIVE-21000
> Project: Hive
> Issue Type: Improvement
> Reporter: Zoltan Haindrich
> Priority: Major
> Attachments: sampler_before.png
>
>
> I was looking into some compile profiles for tables with lots of columns; and it turned out that [thrift 0.9.3 is allocating a List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] during every hashcode calculation; but luckily THRIFT-2877 is improving on that - so I propose to upgrade to at least 0.10.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)