[jira] [Resolved] (HUDI-306) Get to Hudi to support AWS Glue Catalog and other Hive Metastore implementations

["sivabalan narayanan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sivabalan narayanan resolved HUDI-306.
--------------------------------------
    Fix Version/s: 0.5.2
       Resolution: Fixed

> Get to Hudi to support AWS Glue Catalog and other Hive Metastore implementations
> --------------------------------------------------------------------------------
>
>                 Key: HUDI-306
>                 URL: https://issues.apache.org/jira/browse/HUDI-306
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: Hive Integration
>            Reporter: Udit Mehrotra
>            Assignee: Udit Mehrotra
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 0.5.2
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Hudi currently does not work with AWS Glue Catalog. The issue/exception it runs into has been reported here as well [issue|https://github.com/apache/incubator-hudi/issues/954] .
> As mentioned in the issue, the reason for this is:
>  * Currently Hudi is interacting with Hive through two different ways:
>  ** Creation of table statement is submitted directly to Hive via JDBC [https://github.com/apache/incubator-hudi/blob/master/hudi-hive/src/main/java/org/apache/hudi/hive/HoodieHiveClient.java#L472] . Thus, Hive will internally create the right metastore client (i.e. Glue if {{*hive.metastore.client.factory.class*}} is set to {{*com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory*}} in hive-site)
>  ** Whereas partition listing among other things are being done by directly calling hive metastore APIs using hive metastore client: [https://github.com/apache/incubator-hudi/blob/master/hudi-hive/src/main/java/org/apache/hudi/hive/HoodieHiveClient.java#L240]
>  * Now in Hudi code, standard specific implementation of the metastore client (not glue metastore client) is being instantiated: [https://github.com/apache/incubator-hudi/blob/master/hudi-hive/src/main/java/org/apache/hudi/hive/HoodieHiveClient.java#L109] .
>  * Ideally this instantiation of metastore client should be left to Hive through [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L5045] for it to consider other implementations of metastore client that might be configured through {{*hive.metastore.client.factory.class*}} .
> That is the reason that table gets created in Glue metastore, but while reading or scanning partitions it is talking to the local hive metastore where it does not find the table created.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)