You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@gobblin.apache.org by "Zihan Li (Jira)" <ji...@apache.org> on 2020/07/24 21:28:00 UTC

[jira] [Resolved] (GOBBLIN-861) Skip getPartition() call to Hive Metastore when a partition already exists

     [ https://issues.apache.org/jira/browse/GOBBLIN-861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zihan Li resolved GOBBLIN-861.
------------------------------
    Resolution: Fixed

> Skip getPartition() call to Hive Metastore when a partition already exists
> --------------------------------------------------------------------------
>
>                 Key: GOBBLIN-861
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-861
>             Project: Apache Gobblin
>          Issue Type: Task
>          Components: hive-registration
>            Reporter: Zihan Li
>            Assignee: Abhishek Tiwari
>            Priority: Major
>          Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Currently, we compute a diff between the current partition and an already registered partition when a partition has already been registered in Hive. This is done by calling getPartition() on the Hive metastore client, which can be expensive. Since no time-varying attributes are stored in a Hive partition, diff computation (and getPartition() call) can be skipped when a partition already exists.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)