You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Rajesh Balamohan (JIRA)" <ji...@apache.org> on 2016/11/21 12:58:59 UTC
[jira] [Updated] (HIVE-15250) Reuse partitions info generated in
MoveTask to its subscribers (StatsTask)
[ https://issues.apache.org/jira/browse/HIVE-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rajesh Balamohan updated HIVE-15250:
------------------------------------
Attachment: HIVE-15250.1.patch
Observed 10% improvement on top of HIVE-15240 in 5 node cluster running on EC2 in populating web_returns dataset in tpc-ds-200.
> Reuse partitions info generated in MoveTask to its subscribers (StatsTask)
> -----------------------------------------------------------------------------------------
>
> Key: HIVE-15250
> URL: https://issues.apache.org/jira/browse/HIVE-15250
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Rajesh Balamohan
> Priority: Minor
> Attachments: HIVE-15250.1.patch
>
>
> When dynamic partitions are enabled, {{StatsTask}} loads partition information by querying metastore. In cases like {{insert overwrite table}}, this can be expensive operation depending on the number of partitions involved (for e.g, in tpcds populating web_returns table would incur 2184 DB calls just on this function).
> It would be good to pass on the partition information generated in MoveTask to its subscribers to reduce the number of DB calls.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)