You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Sudarshan Vasudevan (Jira)" <ji...@apache.org> on 2020/08/03 19:10:05 UTC
[jira] [Updated] (GOBBLIN-861) Skip getPartition() call to Hive
Metastore when a partition already exists
[ https://issues.apache.org/jira/browse/GOBBLIN-861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sudarshan Vasudevan updated GOBBLIN-861:
----------------------------------------
Fix Version/s: 0.15.0
> Skip getPartition() call to Hive Metastore when a partition already exists
> --------------------------------------------------------------------------
>
> Key: GOBBLIN-861
> URL: https://issues.apache.org/jira/browse/GOBBLIN-861
> Project: Apache Gobblin
> Issue Type: Task
> Components: hive-registration
> Reporter: Zihan Li
> Assignee: Abhishek Tiwari
> Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Currently, we compute a diff between the current partition and an already registered partition when a partition has already been registered in Hive. This is done by calling getPartition() on the Hive metastore client, which can be expensive. Since no time-varying attributes are stored in a Hive partition, diff computation (and getPartition() call) can be skipped when a partition already exists.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)